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FOREWORD 


The Software Engineering Laboratory (SEL) is an organization sponsored by the 
National Aeronautics and Space Administration/Goddard Space Flight Center 
(NASA/GSFC) and created to investigate the effectiveness of software engineering 
technologies when applied to the development of applications software. The SEL was 
created in 1976 and has three primary organizational members: 


NASA/GSFC, Software Engineering Branch 
University of Maryland, Department of Computer Science 
Computer Sciences Corporation, Software Engineering Operation 


The goals of the SEL are (1) to understand the software development process in the GSFC 
environment; (2) to measure the effects of various methodologies, tools, and models on 
this process; and (3) to identify and then to apply successful development practices. The 
activities, findings, and recommendations of the SEL are recorded in the Software 
Engineering Laboratory Series, a continuing series of reports that includes this document. 


The previous version of the Recommended Approach to Software Development was 
published in April 1983. This new edition contains. updated material and constitutes a 
major revision to the 1983 version. The following are primary contributors to the current 
edition: 


Linda Landis, Computer Sciences Corporation 

Sharon Waligora, Computer Sciences Corporation 
Frank McGarry, Goddard Space Flight Center 

Rose Pajerski, Goddard Space Flight Center 

Mike Stark, Goddard Space Flight Center 

Kevin Orlin Johnson, Computer Sciences Corporation 
Donna Cover, Computer Sciences Corporation 


Single copies of this document can be obtained by writing to 
Software Engineering Branch 
Code 552 


Goddard Space Flight Center 
Greenbelt, Maryland 2077 1 
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ABSTRACT 


_ This document presents guidelines for an organized, disciplined approach to software 


development that is based on studies conducted by the Software Engineering Laboratory 
(SEL) since 1976. It describes methods and practices for each phase of a software 
development life cycle that starts with requirements definition and ends with acceptance 


` testing. For each defined life cycle phase, this document presents guidelines for the 


development process and its management, and for the products produced and their reviews. 


| This document is a major revision of SEL-8 1-205. 


NOTE: The material presented in this document is consistent with major NASA/GSFC standards. 


NOTE: The names of some commercially available products cited in this document may be copyrighted or 
registered as trademarks. No citation in excess of fair use, express or implied, is made in this document 


and none should be construed. 
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SECTION 1 


INTRODUCTION 


This document presents a set of guidelines that constitute a disciplined approach to software 
development. It is intended primarily for managers of software development efforts and 
for the technical personnel (software engineers, analysts, and programmers) who are 
responsible for implementing the recommended procedures. This document is neither a 
manual on applying the technologies described here nor a tutorial on monitoring a govern- 
ment contract. Instead, it describes the methodologies and tools that the Software 
Engineering Laboratory (SEL) recommends for use in each life cycle phase to produce 
manageable, reliable, cost-effective software. 


THE FLIGHT DYNAMICS ENVIRONMENT 


The guidelines included here are those that have proved effective in the experiences of the 
SEL (Reference 1). The SEL monitors and studies software developed in support of flight 
dynamics applications at the National Aeronautics and Space Administration/Goddard 
Space Flight Center (NASA/GSFC). Since its formation in 1976, the SEL has collected 
data from more than 100 software development projects. Typical projects range in size 
from approximately 35,000 to 300,000 delivered source lines of code (SLOC) and require 
from 3 to 60 staff-years to produce. 


Flight dynamics software is developed in two distinct computing environments: the Flight 
Dynamics Facility (FDF) and the Systems Technology Laboratory (STL). (See Figure 
1-1.) Mission support software is engineered and operated in the mainframe environment 
of the FDF. This software is used in orbit determination, orbit adjustment, attitude deter- 
mination, maneuver planning, and general mission analysis. Advanced concepts for flight 
dynamics are developed and studied in the STL. Software systems produced in this facility 
include simulators, systems requiring special architectures (e.g., embedded systems), flight 
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Figure 1-1. The SEL Software Engineering Environment 
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dynamics utilities, and projects supporting advanced system studies. 
The STL also hosts the SEL database and the entire set of SEL 
research tools. 


This revised edition of the Recommended Approach to Software 
Development reflects the evolution in life cycle, development 
methodology, and tools that has taken place in these environments in 
recent years. During this time, Ada and object-oriented design 
(OOD) methodologies have been introduced and used successfully. 
The potential for reuse of requirements, architectures, software, and 
documentation has been, and continues to be, studied and exploited. 
Ongoing studies also include experiments with the Cleanroom 
methodology (References 2 through 4), formal inspection, and 
computer-aided software engineering (CASE) tools. 


Because the SEL's focus is process improvement, it is a catalyst for 
this evolution. The SEL continuously conducts experiments using 
the actual, production environment. The lessons learned from these 
experiments are routinely fed back into an evolving set of standards 
and practices that includes the Recommended Approach. 


As these studies are confined to flight dynamics applications, 
readers of this document are cautioned that the guidance presented 
here may not always be appropriate for environments with 
significantly different characteristics. 


DOCUMENT OVERVIEW 


This document comprises 10 sections. Sections 3 through 9 parallel 
the phases of the software development life cycle through acceptance 
testing, and discuss the key activities, products, reviews, 
methodologies, tools, and metrics of each phase. 


Section 1 presents the purpose, organization, and intended 
audience for the document. - 


Section 2 provides an overview of the software development 
life cycle. The general goals of any software development effort 
are discussed, as is the necessity of tailoring the life cycle to adjust 
to projects of varying size and complexity. 


Section 3 provides guidelines for the requirements definition 
phase. Generation of the system and operations concept and the 
requirements and specifications documents are covered. The 
purpose and format of the system concept and requirements reviews 
are outlined. 
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Section 4 discusses the key activities and products of the 
requirements analysis phase: requirements classifications, 
walk-throughs, functional or object-oriented analysis, the 
requirements analysis report, and the software specifications review. 


Section 5 presents the recommended approach to preliminary 
design. The activities, products, and methodologies covered 
include structured and object-oriented design, reuse analysis, design 
walk-throughs, generation of prolog and program design language, 
the preliminary design report, and the preliminary design review. 


Section 6 provides comparable material for the detailed design 
phase. Additional topics include the build test plan, completion of 
prototyping activities, the critical design review, and the detailed 
design document. 


Section 7 contains guidelines for implementation of the 
designed software system. Coding, code reading, unit testing, and 
integration are among the activities discussed. The system test plan 
and user's guide are summarized. 


Section 8 addresses system testing, including test plans, testing 
methodologies, and regression testing. Also covered are preparation 
of the system description document and finalization of the 

acceptance test plan. f 


Section 9 discusses the products and activities of the acceptance 
testing phase: preparing tests, executing tests, evaluating results, 
and resolving discrepancies. 


Section 10 itemizes key DOs and DON'Ts for project success. 


A list of acronyms, references, a 
bibliography of SEL literature, and an 
index conclude this document. 


Although the maintenance and 
Recent SEL papers on software operation phase is beyond the scope of 
maintenance inciude the current document, efforts are now 
am rien ii underway in the SEL to study this important 
rovemen aintenance In A 

the SEL" and "Towards Full part of the life cycle. The results of these 
Life Cycle Control, " both by studies will be incorporated into a future 
Rombach, Ulery, and Valett. edition. 

See References 5 and 6. i 


A A AL 


„A IÀ! 1 


Aa, 


, MRA 


Mt UA 


Alli 


@in 


A 


(TEL E 


A 


UO INpo.jU] - | UOBIES 


| 


Me, 


Section 2 - Life Cycle 


SECTION 2 


THE SOFTWARE DEVELOPMENT LIFE CYCLE 


The flight dynamics software development process is modeled as a series of 
eight sequential phases, collectively referred to as the software development 
life cycle: 


—~ 
2. Requirements Analysis maa 
3. Preliminary Design m 
4. Detailed Design i aS 
S 
[e Syrtem Testing J 
|_7. Acceptance Testing __| S 


Each phase of the software development life cycle is characterized by 
specific activities and the products produced by those activities. 


As shown in Figure 2-1, these eight phases divide the software life cycle 
into consecutive time periods that do not overlap. However, the activities 
characteristic of one phase may be performed in other phases. Figure 2-1 
graphs the spread of activities throughout the development life cycle of 
typical flight dynamics systems. The figure shows, for example, that 
although most of the work in analyzing requirements occurs during the 
requirements analysis phase, some of that activity continues at lower levels 
in later phases as requirements evolve. 
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Example: At the end of the implementation phase (5th dashed line), approximately 46% of the staff are Involved in system testing; = 
approximately 15% are preparing for acceptance testing; approximately 7% are addressing requirements changes or problems; = 
approximately 12% are designing modifications; and approximately 20% are coding, code reading, unit testing, and integrating : 
changes. Data are shown only for the phases of the software life cycie for which the SEL has a representative sample. = 
x 
Figure 2-1. Activities by Percentage of Total Development Staff Effort _ 
= 
= 
PHASES OF THE LIFE CYCLE 
The eight phases of the software development life cycle are defined in the following = 
paragraphs. : 
Requirements Definition = 
Requirements definition is the process by which the needs of the customer are translated 
into a clear, detailed specification of what the system must do. For flight dynamics z 
applications, the requirements definition phase begins as soon as the mission task is 8 
° . i 7 va ce te: . + . E 
established. A team of analysts studies the available information about the mission and $ 
develops an operations concept. This includes a timeline for mission events, required = 
attitude maneuvers, the types of computational processes involved, and specific operational E 
scenarios. The functions that the system must perform are defined down to the level of a i 
subsystem (e.g., a telemetry processor). = 
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In this document, the term analyst refers 
to those specialists in flight dynamics 
(astronomers, mathematicians, physicists, 


and engineers) who determine the detailed 
requirements of the system and perform 
acceptance tests. For these activities, 
analysts work in teams (e.g., the 
requirements definition team} and function 
as agents for the end users of the system. 


In each phase of the life 
cycle, certain milestones 
must be reached in order to 
declare the phase complete. 
Because the life cycle is 
sequential, these exit criteria are also the 
entry criteria for the following phase. In 
this document, entry and exit criteria are 
shown in the summary tables on the first 
page of Sections 3 through 9. A brief 
discussion of the phase's exit criteria is 


Working with experienced developers, analysts 
identify any previously developed software that can 
be reused on the current project. The advantages 
and disadvantages of incorporating the existing 
components are weighed, and an overall 
architectural concept is negotiated. The results of 
these analyses are recorded in the system and 
operations concept (SOC) document and assessed in 
the system concept review (SCR). 


Guided by the SOC, a requirements definition team 
derives a set of system-level requirements from 
documents provided by the mission project office. 
A draft version of the requirements is then recast in 
terms suitable for software design. These 
specifications define what data will flow into the 
system, what data will flow out, and what steps 
must be taken to transform input to output. 
Supporting mathematical information is included, 
and the completed requirements and specifications 
document is published. The conclusion of this 


i 


provided at the conclusion of each section, 


phase is marked by the system requirements review 
(SRR), during which the requirements and 
specifications for the system are evaluated. 


Requirements Analysis- 


The requirements analysis phase begins after the SRR. In this 
phase, the development team analyzes the requirements and 
specifications document for completeness and feasibility. The 
development team uses structured or object-oriented analysis and a 
requirements classification methodology to clarify and amplify the 
document. Developers work closely with the requirements 
definition team to resolve ambiguities, discrepancies, and to-be- 
determined (TBD) requirements or specifications. 


The theme of reuse plays a prominent role throughout the 
requirements analysis and design phases. Special emphasis is 
placed on identifying potentially reusable architectures, designs, 
code, and approaches. (An overview of reuse in the life cycle is 
presented later in this section.) 


When requirements analysis is complete, the development team 
prepares a summary requirements analysis report as a basis for 
preliminary design. The phase is concluded with a software 
specifications review (SSR), during which the development team 
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presents the results of their analysis for evaluation. The 
requirements definition team then updates the requirements and 
specifications document to incorporate any necessary modifications. 


Preliminary Design 


The baselined requirements and specifications form a contract 
between the requirements definition team and the development team 
and are the starting point for preliminary design. During this phase, 
members of the development team define the software architecture 
that will meet the system specifications. They organize the 
requirements into major subsystems and select an optimum design 
from among possible alternatives. All internal and external 
interfaces are defined to the subsystem level, and the designs of 
high-level functions/objects are specified. 


The development team documents the high-level design of the 
system in the preliminary design report. The preliminary design 
phase culminates in the preliminary design review (PDR), where the 
development team formally presents the design for evaluation. 


Detailed Design 


During the detailed design phase, the development team extends the 
software architecture defined in preliminary design down to the unit 
level. By successive refinement techniques, they elaborate the 
preliminary design to produce "code-to" specifications for the 
software. All formalisms for the design are produced, including the 
following: 


¢ Functional or object-oriented design diagrams 

¢ Descriptions of all user input, system output (for example, 
screen, printer, and plotter), and input/output files 

* Operational procedures 

e Functional and procedural 
descriptions of each unit 

¢ Descriptions of all internal interfaces 


among units - 
DEFINITIONS 


The development team documents these 
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Throughout this document, the term 


design specifications in the detailed design unit is used to designate any set of 
document that forms the basis for program statements that are logically 

A f me . : treated as a whole. A main program, a 
implementation. At the critical design review subroutine, or a subprogram may each 
(CDR), which concludes this phase, the Se ely 7 roe A moules a 
detailed design is examined to determine Componente Saints Eaulish: 


language sense to denote any 


whether levels of detail and completeness are ë 
constituent part. 


sufficient for coding to begin. 
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Implementation 


In the implementation (code, unit testing, and integration) phase, the 
developers code new components from the design specifications and 
revise existing components to meet new requirements. They 
integrate each component into the growing system, and perform unit 
and integration testing to ensure that newly added capabilities 
function correctly. 


In a typical project, developers build several subsystems 
simultaneously from individual components. The team repeatedly 
tests each subsystem as new components are coded and integrated 
into the evolving software. At intervals, they combine subsystem 
capabilities into a complete working system for testing end-to-end 
processing capabilities. The sequence in which components are 
coded and integrated into executable subsystems and the process of 
combining these subsystems into systems are defined in an 
implementation plan that is prepared by development managers 
during the detailed design phase. 


The team also produces a system test plan and a draft of the user's 
guide in preparation for the system testing phase that follows. 
Implementation is considered complete when all code for the system 
has been subjected to peer review, tested, and integrated into the 
system. 


System Testing 


During the system testing phase, the development team validates the 
completely integrated system by testing end-to-end capabilities 
according to the system test plan. The system test plan is based on 
the requirements and specifications document. Successfully 
completing the tests specified in the test plan demonstrates that the 
system satisfies the requirements. 


In this phase, the developers correct any errors uncovered by system 
tests. They also refine the draft user's guide and produce an initial 
system description document. System testing is complete when all 
tests specified in the system test plan have been run successfully. 


Acceptance Testing 


In the acceptance testing phase, the system is tested by an 
independent acceptance test team to ensure that the software meets 
all requirements. Testing by an independent team (one that does not 
have the developers’ preconceptions about the functioning of the 
system) provides assurance that the system satisfies the intent of the 
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original requirements. The acceptance test team usually consists of 
analysts who will use the system and members of the requirements 
definition team. 


The tests to be executed are specified in the acceptance test plan 
prepared by the acceptance test team before this phase. The plan is 
based on the contents of the requirements and specifications 
document and approved specification modifications. 


During acceptance testing, the development team assists the test team 
and may execute acceptance tests under its direction. Any errors 
uncovered by the tests are corrected by the development team. 
Acceptance testing is considered complete when the tests specified in 
the acceptance test plan have been run successfully and the system 
has been formally accepted. The development team then delivers 
final versions of the software and the system documentation (user's 
guide and system description) to the customer. 


Maintenance and Operation 


At the end of acceptance testing, the system becomes the 
responsibility of a maintenance and operation group. The activities 
conducted during the maintenance and operation phase are highly 
dependent on the type of software involved. For most flight 
dynamics software, this phase typically lasts the lifetime of a 
spacecraft and involves relatively few changes to the software. For 
tools and general mission support software, however, this phase 
may be much longer and more active as the software is modified to 
respond to changes in the requirements and environment. 


The maintenance and operation phase is not 
specifically addressed in this document. 
However, because enhancements and error 
corrections also proceed through a 


Recent SEL studies have shown that most 
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of the effort in initial maintenance of flight 
dynamics systems is spent in enhancing 
the system after launch to satisfy new 
requirements for long-term operational 
support. Such enhancements are usually 
effected without radically altering the 
architecture of the system. Errors found 
during the maintenance and operation 
phase are generally the same type of faults 


development life cycle, the recommended 
approach described here is, for the most 
part, applicable to the maintenance and 
operation phase. The number and 
formality of reviews and the amount of 
documentation produced during 


el 


| 


maintenance and operation vary depending 
on the size and complexity of the software 
and the extent of the modifications. 
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as are uncovered during development, 
although they require more effort to repair. 
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TAILORING THE LIFE CYCLE 


One of the key characteristics that has shaped the SEL's 
recommended approach to software development is the homo- 
geneous nature of the problem domain in the flight dynamics 
environment. Most software is designed either for attitude 
determination and control for a specific mission, for mission-general 
orbit determination and tracking, or for mission planning. These 
projects progress through each life cycle phase sequentially, 
generating the standard documents and undergoing the normal set of 
reviews. 


Certain projects, however, do not fit this mold. 
Within the STL, experiments are conducted to 
study and improve the development process. 
Advanced tools are developed. For these 


development efforts — prototypes, expert 

The software development/ systems, database tools, Cleanroom 

management plan (Soh muat experiments, etc. — the life cycle and the 
scri ow the life cycle will be ; n r 

tailored for a specific project. See methodologies it incorporates often „need 

Section 4 for more details. adjustment. Tailoring allows variation in the 


level of detail and degree of formality of 
documentation and reviews, which may be 
modified, replaced, or combined in the tailoring 
process. Such tailoring provides a more exact 
mätch to unique project requirements and 
development products at a lower overall cost to 
the project without sacrificing quality. 


The following paragraphs outline general guidelines for tailoring the 
life cycle for projects of varying size and type. Additional 
recommendations may be found throughout this document, 
accompanying discussions of specific products, reviews, methods, 
and tools. 


Builds and Releases 


The sizes of typical flight dynamics projects vary considerably. 
Simulators range from approximately 30 thousand source lines of 
code (KSLOC) to 160 KSLOC. Attitude ground support systems 
for specific missions vary between 130 KSLOC and 300 KSLOC, 
while large mission-general systems may exceed 1 million SLOC. 
The larger the project, the greater the risk of schedule slips, 
requirements changes, and acceptance problems. To reduce these 
risks, the implementation phase is partitioned into increments 
tailored to the size of the project. 
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Flight dynamics projects with more than 10 
KSLOC are implemented in builds. A build isa 
portion of a system that satisfies, in part or 
completely, an identifiable subset of the 
specifications. Specifications met in one build 
also are met in all successor builds. The last 
build, therefore, is the complete system. 


A release is a build that is delivered for 
acceptance testing and subsequently released for 
operational use. Projects of fewer than 300 
KSLOC are usually delivered in a single release, 


Reviews are recommended 
for each build. The 
suggested format and 
contents of build design 
reviews are provided in 
Section 7. 


unless otherwise dictated by scheduling (e.g., Guidelines for tailoring the 


launch) considerations or by TBD requirements. 
Large projects (more than 300 KSLOC) are 
generally delivered in multiple releases of 300 to 


development approach (including 
reviews, documentation, and testing) 
for projects of differing scope and 
function are provided throughout this 


500 KSLOC each. document. Look for the scissors 


symbol in the margin. 


Builds within large projects may last up to 6 
months. Builds within small projects may be 
only 2 to 3 months in duration. 


Reviews 


Reviews are conducted to ensure that analysts and developers 
understand and fulfill customer needs. Because reviews are 
designed to assist developers, not to burden them unnecessarily, the 
number of reviews held may vary from project to project. For tools 
development, the requirements, requirements analysis, and 
preliminary design might be reviewed together at PDR. For small 
projects spanning just several months, only two reviews may be 
applicable — the SRR and CDR. For very large projects, a CDR 
could (and should) be held for each major release and/or subsystem 
to cover all aspects of the system and to accommodate changing 
requirements. 


The criteria used to determine whether one or more reviews can be 

combined depend on the development process and the life cycle 

phase. In the requirements analysis phase, for example, answers to 

the following questions would help determine the need for a separate 

SSR: 

e Are there outstanding analysis issues that need to be 
reviewed? 

e How much time will there be between the start of 
requirements analysis and the beginning of design? 

* How stable are the requirements and specifications? 
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On small projects, technical reviews can be no more formal than a 
face-to-face meeting between the key personnel of the project and 
the customer technical representative. On typical flight dynamics 
projects, however, reviews are formalized and follow specific 
formats. Guidelines for these reviews are provided in Sections 3 
through 9. 


Documentation 


On small projects, technical documentation is less formal than on 
medium or large projects, and fewer documents are published. 
Documents that would normally be produced separately on larger 
projects are combined. On a small research project, a single design 
document may replace the preliminary design report, detailed design 
document, and system description. 


Testing and Verification 


Independent testing is generally not performed on small-scale, tool- 
development efforts. Test plans for such projects can be informal. 
Although code reading is always performed on even the smallest 
project, units are often tested in logically related groups rather than 
individually, and inspections are usually conducted in informal, one- 
on-one sessions. 


Configuration Management and Quality Assurance 


Configuration management encompasses all of the activities 
concemed with controlling the contents of a software system. These 
activities include monitoring the status of system components, 
preserving the integrity of released and developing versions of a 
system, and governing the effects of changes throughout the 
system. Quality assurance activities ensure that software 
development processes and products conform to established 
technical requirements and quality standards. 


All software and documentation that are developed for delivery are 
generally subject to formal configuration management and quality 
assurance controls. Tools developed exclusively for internal use are 
exempt, unless the tool is required to generate, run, or test a 
deliverable system. 


On medium and small projects, configuration control may be 
performed by a designated member of the development team — a 
practice that is strongly discouraged on a large project. Similarly, 
the quality assurance function may be assigned to a team member 
with other responsibilities or may be handled by the technical leader. 
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Prototyping 


A prototype is an early experimental model of a system, system 
component, or system function that contains enough capabilities for 
it to be used to establish or refine requirements or to validate critical 
design concepts. In the flight dynamics environment, prototypes are 
used to (1) mitigate risks related to new technology (e.g., hardware, 
language, design concepts) or (2) resolve requirements issues. In 
the latter case, entire projects may be planned as prototyping efforts 
that are designed to establish the requirements for a later system. 


Unless the end product of the entire project is 
a prototype, prototyping activities are usually 
completed during the requirements analysis 
and design phases. The prototyping activity 
has its own, usually informal, life cycle that is 


All prototyping activities must be 


embedded within the early phases of the full planned and controlled. The plan 

system's life cycle. If any portion of the must define the purpose and scope of 
: ee the prototyping effort, and must 

prototype is to become part of the final establish specific completion criteria. 


system, it must be validated through all the See Section 4 for more details. 
established checkpoints (design reviews, code 
reading, unit testing and certification, etc.). 
As a tule, such prototyping activities should 
require no more than 15 percent of the total 
development effort. 


WHEN TO PROTOTYPE 


Asa rule of thumb, use prototyping 

whenever 

+ the project involves new technology, 
e.g., new hardware, development 
language, or system architecture 

* the requirements are not understood 

. there are major, unresolved issues 
concerning performance, reliability, 
or feasibility 

« the user interface is critical to system 

success or is not clearly understood 


For projects in which the end product is a 
prototype, however, an iterative life cycle may 
be preferable. This is particularly true when a 
new user interface is a significant component 
of the system. An initial version of the 
prototype is designed, implemented, and 
demonstrated to the customer, who adds or 
revises requirements accordingly. The 
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prototype is then expanded with additional 
builds, and the cycle continues until 
completion criteria are met. 


Tailoring the life cycle for any type of prototyping requires careful 
planning. The more new technology that is to be used on a project, 
the greater the prototyping effort. The larger the prototyping effort, 
the more formalized must be its planning, development, and 
management. The results of even the smallest prototyping effort 
must always be documented. Lessons learned from the prototype 
are incorporated into plans for subsequent phases and are included 
in the project history. See Section 4 for additional guidance on 
planning and documenting prototyping activities. 
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KEY REUSE ELEMENTS 


e software architecture 


¢ software development process 


» design architecture or 
concepts 


* test plans and procedures 


* code 
«user documentation 
« staff 


domain 


analysis 


requirements 
generalization 


Analyze these key elements of a 
project for possible reuse: 
* requirements characteristics 


REUSE THROUGHOUT THE LIFE CYCLE 


From the beginning to the end of the life cycle, the approach to 
software development recommended by the SEL stresses the 
principle of reuse. Broadly speaking, the reuse of existing 
experience is a key ingredient to progress in any area. Without 
reuse, everything must be relearned and re-created. In software 
development, reuse eliminates having to "reinvent the wheel" in each 
phase of the life cycle, reducing costs and improving both reliability 
and productivity. 


Planning for reuse maximizes these benefits by allowing the cost of 
the learning curve in building the initial system to be amortized over 
the span of follow-on projects. Planned reuse is a primary force 
behind such recent technologies as object-oriented design and Ada. 


All experience and products of the software 
development life cycle — specifications, designs, 
documentation, test plans, as well as code — have 
potential for reuse. In the flight dynamics 
environment, particular benefits have been 
obtained by reusing requirements and specifi- 
cations (i.e., formats, key concepts, and high- 
level functionality) and by designing for reuse (see 
References 7 through 10). 


Figure 2-2 shows how reuse activities fit into the 
software development life cycle. The top half of 
the figure contains activities that are conducted to 
enable future reuse. The lower half shows 
activities in which existing software is used in the 
system under development. These activities are 
outlined in the following paragraphs. 


Activities That Enable Future Reuse 


Domain analysis is the examination of the application domain of the 
development organization to identify common requirements and 
functions. It is usually performed during the requirements definition 
and analysis phases, but it may also be conducted as a separate 
activity unconnected to a particular development effort. Domain 
analysis produces a standard, general architecture or model that 
incorporates the common functions of a specific application area and 
can be tailored to accommodate differences between individual 
projects. It enables requirements generalization, i.e., the 
preparation of requirements and specifications in such a way that 
they cover a selected "family" of projects or missions. 
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Figure 2-2. Reuse Activities Within the Life Cycle 


Software not originally intended for reuse is more difficult to 
incorporate into a new system than software explicitly designed for 
reuse. Designing for reuse provides modularity, standard inter- 
faces, and parameterization. Design methods that promote 
reusability are described in References 9 and 11. 


Reuse libraries hold reusable source code and associated 
requirements, specifications, design documentation, and test data. 
In addition to storing the code and related products, the library 
contains a search facility that provides multiple ways of accessing 
the software (e.g., by keyword or name). On projects where reuse 
has been a design driver, extraction of candidate software for 
inclusion in the reuse library takes place after system testing is 
complete. 


Reuse on Current Projects 


During the requirements definition and analysis phases, reuse 
analysis is performed to determine which major segments 
(subsystems) of existing software can be used in the system to be 
developed. In the design phases, developers verify this analysis by 
examining each reusable element individually. During the 
preliminary design phase, developers evaluate major components to 
determine whether they can be reused verbatim or must be modified; 
individual units from the reuse library are examined during the 
detailed design phase. 
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Section 2 - Life Cycle 


Software may be reused verbatim or may be modified to fit the 
needs of the current project. During the implementation phase, 
developers integrate existing, unchanged units into the developing 
system by linking directly to the reuse library. Modified software, 
on the other hand, must be subjected to peer review and unit testing 
before being integrated. 


A final reuse activity takes place during the maintenance and 
operation phase of the life cycle. Through the changes that it 
implements, the maintenance team can positively or negatively affect 
the reusability of the system; “quick fixes", for example, may 
complicate future reuse. Reuse preservation techniques for 
maintenance use many of the same practices that promote reuse 
during the analysis, design, and implementation phases. 


reuse 
preservation 


MEASURES 


Measures of project progress and viability are key to the effective 

management of any software development effort. In each phase of 

the life cycle, there are certain critical metrics that a manager must 

examine to evaluate the progress, stability, and quality of the 
development project. 


Both objective and subjective data are 
measured. Objective data are actual counts of 
items (e.g., staff hours, SLOC, errors) that can 
be independently verified. Subjective data are 


pire edema ist dependent on an individual's or group's 
detailed information assessment of a condition (e.g., the level of 
ABOU NEPCA í difficulty of a problem or the clarity of 
Shasa Look Tar tho” requirements). Together, these data serve as a 
MEASURES heading and system of checks and balances. Subjective data 
symbol: provide critical information for interpreting or 


validating objective data, while objective data 
provide definitive counts that may cause the 
manager to question his or her subjective 
understanding and to investigate further. 


Objective measures can be further classified into two groups: those 
that measure progress or status and those that measure project 
quality (e.g., stability, completeness, or reliability). Progress 
measures, such as the number of units coded or the number of tests 
passed, are evaluated against calculations of the total number of 
items to be completed. Quality measures, on the other hand, are 
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Table 2-1. Measures Recommended by the SEL 


MEASURES FREQUENCY MAJOR APPLICATION 
CLASS 


Estimates of: Managers Monthly e Project stability 
*Total SLOC + Planning aid 
STATUS 


(new, modified, 
ERRORS/ 


reused) 
CHANGES 


* Total units 
FINAL 
CLOSE-OUT 


* Total effort 
+ Major dates 

only useful if the manager has access to models or metrics that represent what should be 

expected. 


» Staff hours Developers Weekly + Project stability 
{total & by activity) « Replanning indicator 
» Computer use Automated Weekly « Effectiveness/impact of the 


development process 
being applied 


tool 


+ Project progress 

+ Adherence to defined 
process 

+ Stability and quality of 
requirements 


» Requirements 
(growth, TBDs, 
changes, Q&As) 

* Units designed, 
coded, tested 

* SLOC (cumulative) 

* Tests (complete, 

passed) 


Biweekly 


Managers 


Developers Biweekly 


Weekly 
Biweekly 


Automated 
Developers 


+ Effectiveness/impact of the 
development process 

« Adherence to defined 

process 


¢ Errors (by Developers By event 
category) 

« Changes (by 
category} 


* Changes (to source) 


Developers By event 


Automated Weekly 


* Build predictive models 
+ Plan/manage new projects 


itime, at 
completion 


Actuals at completion: 
: Effort 
. Size (SLOC, 
units) 
+ Source 
characteristics 
+ Major dates 


Managers 


In the SEL, measurement data from current and past projects are stored in a project 
histories database. Using information extracted from such a database, managers can gauge 
whether measurement trends in the current project differ from the expected models for the 
development environment. (See Section 6 of Reference 12.) 


The management measures recommended by the SEL are listed in Table 2-1. Figure 2-3 
shows in which phases of the life cycle each of these measures is collected. 


As Table 2-1 shows, developers are responsible for providing many of the measures that 
are collected. In the SEL, developers use various data collection forms for this purpose. 
The individual forms are discussed in the sections of this document covering the life-cycle 
phases to which they apply. 
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Figure 2-3. Graph Showing in Which Life-Cycle Phases Each Measure Is Collected 


EXPERIMENTATION 


Measurement is not only essential to the management of a software 
development effort; it is also critical to software process 
improvement. In the SEL, process improvement is a way of life. 
Experiments are continually being conducted to investigate new 
software engineering technologies, practices, and tools in an effort 
to build higher-quality systems and improve the local production 
process. The SEL's ongoing measurement program provides the 
baseline data and models of the existing development environment 
against which data from experimental projects are compared. 


For several years, the SEL has been conducting experiments and 
measuring the impact of the application of the Cleanroom 
methodology (References 2, 3, and 4), which was developed in the 
early 1980s by Harlan Mills. The goal of the Cleanroom 
methodology is to build a software product correctly the first time. 
Cleanroom stresses disciplined "reading" techniques that use the 
human intellect to verify software products; testing is conducted for 
the purpose of quality assessment rather than as a method for 
detecting and repairing errors. 


The Cleanroom methodology is still in the early stages of evaluation 
by the SEL. Although some of the methods of Cleanroom are the 
same as existing methods in the SEL's recommended approach — 
e.g., code reading — other aspects remain experimental. 
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Consequently, the Cleanroom methodology is used throughout this 
document as an example of the integral aspect of experimentation 
and process improvement to the SEL's recommended approach. 
Variations in life cycle processes, methods, and tools resulting from 
the application of Cleanroom will be highlighted. Look for the 
experimentation symbol. 


DEFINITION 


The term Cleanroomwas 
borrowed from integrated circuit 
production. It refers to the 


dust-free environments in which 
the circuits are assembled. 
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UFE 
REQUIREMENTS 
CYCLE DEFINITION N 


SECTION 3 


THE REQUIREMENTS DEFINITION PHASE 


ENTRY CRITERIA 


¢ Problenvproject description completed 
e Project approved 


* System and operations concept document f 


* Requirements and specifications 
document 


e Staff hours 

. Number of requirements defined 
vs. estimated total requirements 

« Percentage of requirements with 
completed specifications 


¢ Structured or object-oriented analysis 
« Walk-throughs 
e Prototyping 


EXIT CRITERIA 


« System and operations concept completed 


___ SRR completed 


+ Requirements and specifications baselined 


eats 4 


KEY ACTIVITIES 


Requirements Definition Team 
» Develop a system concept 
* Prepare the reuse proposal 
+ Develop an operations concept 
* Define the detailed requirements 
* Derive the specifications 
* Conduct the SCR and SRR 


Management Team 
* Develop a plan for the phase 
* Staff and train the requirements 
definition team 
* Interact with the customer 
e Evaluate progress and products 
* Control major reviews 


21 


Section 3 - Requirements Definition 


OVERVIEW 


The purpose of the requirements definition phase is to produce a 
clear, complete, consistent, and testable specification of the technical 
requirements for the software product. 


Requirements definition initiates the software development life 
cycle. During this phase, the requirements definition team uses an 
iterative process to expand a broad statement of the system 
requirements into a complete and detailed specification of each 
function that the software must perform and each criterion that it 
must meet. The finished requirements and specifications, combined 
with the system and operations concept, describe the software 
product in sufficient detail so that independent software developers 
can build the required system correctly. 


The starting point is usually a set of high-level requirements from 
the customer that describe the project or problem. For mission 
support systems, these requirements are extracted from project 
documentation such as the system instrumentation requirements 
document (SIRD) and the system operations requirements document 
(SORD). For internal tools, high-level requirements are often 
simply a list of the capabilities that the tool is to provide. 


In either case, the requirements definition team formulates an overall 
concept for the system by examining the high-level requirements for 
similarities to previous missions or systems, identifying existing 
software that can be reused, and developing a preliminary system 
architecture. The team then defines scenarios showing how the 
system will be operated, publishes the system and operations 
concept document, and conducts a system concept review (SCR). 
(See Figure 3-1.) 


Following the SCR, the team 
NOTE icont.) 


derives detailed requirements 
for the system from the high- 
level requirements and the 
system and operations concept. 
Using structured or object- 
oriented analysis, the team 
specifies the software functions 
and algorithms needed to satisfy 
each detailed requirement. 
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analysts from the 
requirements 
definition team plan 
acceptance tests, 
and that developers 
assist in defining 
requirements, 
planning for reuse, 
and supporting 
acceptance testing. 


In the flight dynamics 
environment, 
membership in 
the teams that 
perform the 
technical activi- 
ties of software 
development overlaps 
The overlap ensures 
that experienced... 
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PROJECT OR PROBLEM 
DESCRIPTION 


REQUIREMENTS gg aan 
usT ITEMIZED HGH-LEVEL 


ruNcnoxs TO mi REQUIREMENTS 
REQUIREMENTS 
TRACE 


p ed 


SYSTEM AND OPERATIONS 
CONCEPT DOCUMENT 
ENGINEERING 
STUDY REPORTS 


REUSE PROPOSAL 


HIGH-LEVEL SYSTEM 
ARCHITECTURE 


OPERATIONS SCENARIOS 


{REFINED) 


DOCUMENTATION FROM 
PREVIOUS PROJECTS OPERATIONS 


ed CONCEPTS 


3.3 


RIDS AND 
RESPONSES 


SCR 
PRESENTATION 
SCR 
PARTICIPA! 


SCR-HARDCOPY-MATERIALS 


NOTE: In this figure, as in all data flow diagrams (DFDs) in this document, rectangles denote external entities, 
circles represent processes, and parallel lines are used for data stores (in this case, documents). The processes 
labelled 3.1, 3.2, and 3.3 are described in the KEY ACTIVITIES subsection below. The SCR is described under 


REVIEWS and the system and operations concept document is covered in PRODUCTS. 


Figure 3-1. Generating the System and Operations Concept 


When the specifications are complete, the requirements definition team publishes the 
requirements and specifications document in three parts: (1) the detailed requirements, (2) 
the functional or object-oriented specifications, and (3) any necessary mathematical 
background information. At the end of the phase, the team conducts a system requirements 
review (SRR) to demonstrate the completeness and quality of these products. (See Figure 
3-2.) 
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SYSTEM AND OPERATIONS 
CONCEPT DOCUMENT 
PROJECT OR PROBLEM b 
DESCRIPTION 


INFORMATION FROM 
DEFINE DETAILED PREVIOUS PROJECTS 
REQUIREMENTS 


ITEMIZED HIGH-LEVEL REQUIREMENTS DaT 
YeraiLeo 


REQUIREMENTS 
INTERFACE CONTROL DOCUMENTS 


SPECIFICATIONS 
TRACEABILITY 


MATRI 
DEVELOP 


SPECIFICATIONS 
37 


REQUIREMENTS AND SPECIFICATIONS 
SPECIFICATIONS DOCUMENT 


MATH BACKGROUND 


CONDUCT 
SYSTEM 
RIDS AND REQUIREMENTS 
RESPONSES REVIEW 


SRA 3.8 


PRESENTATION 


SRR 


PARTICIPANTS SAR HARDCOPY MATERIALS 


NOTE: The processes labelled 3.5, 3.6, and 3.7 are discussed in the KEY ACTIVITIES subsection. The requirements and 
specifications document is described under the heading PRODUCTS. The REVIEWS subsection covers the SRR. 


Figure 3-2. Developing Requirements and Specifications 
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KEY ACTIVITIES 


The key technical and managerial activities of the requirements 
definition phase are itemized below. 


Activities of the Requirements Definition Team 


* Develop a system concept. Collect and itemize all high-level 
requirements for the system. Describe the basic functions that 
the system must perform to satisfy these high-level 
requirements. Address issues such as system lifetime (usage 
timelines), performance, security, reliability, safety, and data 
volume. 


From this functional description, generate an ideal, high-level 
system architecture identifying software programs and all major 
interfaces. Allocate each high-level requirement to software, 
hardware, or a person. Specify the form (file, display, printout) 
of all major data interfaces. 


* Prepare the reuse proposal. Review the 
requirements and specifications, system 
descriptions, user's guides, and source code of 
related, existing systems to identify candidates for 
reuse. For flight dynamics mission support 
systems, this involves reviewing support systems 
for similar spacecraft. Select strong candidates 

often combined into a single and estimate the corresponding cost and reliability 
phase: On auch projec: benefits. Determine what compromises are 
developers generally perform 

all requirements definition necessary to reuse software and analyze the 
activities. tradeoffs. 


TAILORING NOTE 


On small projects that are 
developing tools or 

prototypes, requirements 
definition and analysis are 


Adjust the high-level architecture to account for 
reuseable software. Record the results of all reuse 
analysis in a reuse proposal that will be included 
REUSE NOTE in the system and operations concept document. 

Although use of existing software can -* Develop an operations concept. This clearly 


reduce effort significantly, some , A E 
compromises may be necessary. Ensure defines how the system must operate within its 


that all tradeoffs are well understood. 
Avoid these two pitfalls: 

+ Failing to make reasonable compromises, 
thus wasting effort for marginal 
improvement in quality or functionality 

+ Making ill-advised compromises that 
save development effort at the cost of 
significantly degrading functionality or 
performance 


environment. Include operational scenarios for all 
major modes of operation (e.g., emergency versus 
normal). Be sure to include the end-user in this 
process. Conduct an SCR. 


25 


Section 3 - Requirements Definition 


26 


Define the detailed 
requirements. Based on 
the high-level requirements 
and the system concept and 
architecture, define all 
software requirements down 
to the subsystem level. If 
the system is large (with 
many subsystems) or if it 
will interface with other 
systems, explicitly define all 
external interfaces. 


See the PRODUCTS 
subsection below for 


Determine system performance and reliability requirements. If 


certain acceptance criteria apply to a requirement (e.g., meeting a 
particular response time), specify the test criteria with the 
requirement. Identify all intermediate products needed to 
acceptance test the system. 


Derive the functional specifications for the system from the 
requirements. Identify the primary input and output data needed 
to satisfy the requirements. Use structured or object-oriented 
analysis to derive the low-level functions and algorithms the 
software must perform. Define all reports and displays and 
indicate which data the user must be able to modify. 


Keep the specifications design-neutral and language-neutral; i.e., 
concentrate on what the software needs to do, rather than how it 
will do it. Create a traceability matrix to map each low-level 


detailed contents of the 


system and operations 
concept as well as the 
requirements and 

functional specifications 
documents. 


The SCR and SRR 
are covered in 
detail in the 
REVIEWS 
subsection. 


i 


function or data specification to the requirements it fulfills. 


Ensure that all requirements and 
specifications are given a thorough peer 
review. Watch for interface problems 
among major functions and for 
specifications that are duplicated in 
multiple subsystems. Ensure 
compatibility and consistency in 
notation and level of accuracy among 
the specified algorithms. 


Prepare the requirements and 
specifications document, including any 
necessary mathematical background 
information, as a basis for beginning 
software development. 


TAILORING NOTE 


On very large or complex projects, 
it is generally advisable to hold a 
preliminary system requirements 
review (PSRR) as soon as a draft of 
the requirements document is 
complete. This allows end-users 
and key developers to raise critical 
issues before requirements are 
finalized. See the REVIEWS 
subsection for additional 
information on the PSRR. 
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* Conduct the SRR and incorporate approved changes into the 
requirements and specifications. Place the document under 
configuration management as the system baseline. 


Activities of the Management Team 


The management activities performed during this phase pave the 
way for all future phases of the project's life cycle. Specifically, 
managers must accomplish the following: 


* Develop a plan for the phase. (Detailed planning of the entire 
development effort is deferred to the requirements analysis 
phase, after system specifications have been defined.) Address 
the staffing of the teams that will perform the technical work, the 
groups and individuals that will interface with the teams, the 
technical approach, milestones and schedules, risk management, 
and quality assurance. List the reviews to be conducted and 
their level of formality. 


* Staff and train the requirements definition team. Ensure that 
the team contains the necessary mix of skills and experience for 
the task. For mission support systems, the team should include 
analysts with strong backgrounds in mission analysis, attitude 
and orbit determination, and operations. The reuse working 
group must include key software developers as well as 
experienced analysts. Ensure that staff members have the 
necessary training in the procedures, methods, and tools needed 
to accomplish their goals. 


* Interact with the customer to assure 
visibility and resolution of all issues. 
Conduct regular status meetings and ensure 


DEFINITION 


The key developers who participate in 
reuse analysis and other requirements 
definition activities have special technical 
roles throughout the life cycle. The value 
of these application specialists lies in their 
spacific knowledge and experience. On 
mission support projects, for example, the 
application specialist will not only have 
developed such software previously, but 
also will understand the complex mathe- 
matics and physics of flight dynamics. The 
application specialist often acts as a 
“translator,” facilitating communications 
between analysts and developers. 


communications among team members, 
managers, customers, and other groups 
working on aspects of the project. 


Evaluate progress and products. Review 
the system and operations concept and the 
requirements and specifications. Collect 
progress measures and monitor adherence 
to schedules and cost. 


Control major reviews. Ensure that key 
personnel are present at reviews, both 
formal and informal. Participate in the 
SCR and SRR. 
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METHODS AND TOOLS 


The methods and tools used during the requirements definition 
phase are 


+ Structured or object-oriented analysis 
+ Walk-throughs 
¢ Prototyping 


Each is discussed below. 


Analysis Methodologies 


Structured analysis and object-oriented analysis are techniques used 
to understand and articulate the implications of the textual statements 
found in the requirements definition. The requirements definition 
team uses analysis techniques to derive the detailed specifications for 
the system from the higher-level requirements. The analysis 
methodology selected for the project should be appropriate to the 
type of problem the system addresses. 


Functional decomposition is currently the most commonly used 
method of structured analysis. Functional decomposition focuses 
on processes, each of which represents a set of transformations of 
input to output. Using this method, the analyst separates the 
primary system function into successively more detailed levels of 
processes and defines the data flows between these processes. 
Authors associated with structured analysis include E. Yourdon, 
L. Constantine, and T. DeMarco (References 13 and 14). S. 
Mellor and P. Ward have published a set of real-time extensions to 
this method for event-response analysis (Reference 15). 


Object-oriented analysis combines techniques from the realm of data 
engineering with a process orientation. This method defines the 
objects (or entities) and attributes of the real-world problem domain 
and their interrelationships. The concept of an object provides a 
means of focusing on the persistent aspect of entities — an emphasis 
different from that of structured analysis. An object-oriented 
approach is appropriate for software designed for reuse because 
specific objects can be readily extracted and replaced to adapt the 
system for other tasks (e.g., a different spacecraft). Details of the 
object-oriented approach may be found in References 11, 16, and 
17. 

In structured analysis, functions are grouped together if they are 
steps in the execution of a higher-level function. In object-oriented 
analysis, functions are grouped together if they operate on the same 
data abstraction. Because of this difference, proceeding from 
functional specifications to an object-oriented design may necessitate 
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recasting the data flow diagrams. This is a 
significant amount of effort that can be avoided 
by assuming an object-oriented viewpoint 
during the requirements definition phase. 


CASE tools can greatly increase The diagramming capabilities of CASE tools 


productivity, but they can only aid or facilitate application of the chosen analysis 
improve those activities that the 


team or individual knows how to 


methodology. The tools provide a means of 


perform manually. CASE tools producing and maintaining the necessary data 


cannot improve analysis, qualify 
designs or code, etc., if the user does 


flow and object-diagrams online. They usually 


not have have a clear definition of the include a centralized repository for storing and 
manualiprocess involved, retrieving definitions of data, processes, and 


walk-throughs 


inspections 


entities. Advanced tools may allow the 
specifications themselves to be maintained in 
the repository, making it easier to trace the 
requirements to design elements. 


Selected tools should be capable of printing the diagrams in a form 
that can be directly integrated into specifications and other 
documents. Examples of CASE tools currently used in the flight 
dynamics environment include System Architect and Software 
Through Pictures. 


Walk-throughs 


In all phases of the life cycle, peer review ensures the quality and 
consistency of the products being generated. The SEL recommends 
two types of peer review — walk-throughs and inspections — in 
addition to formal reviews such as the SRR and CDR. 


Walk-throughs are primarily conducted as an aid to understanding, 
so participants are encouraged to analyze and question the material 
under discussion. Review materials are distributed to participants 
prior to the meeting. During the meeting, the walk-through leader 
gives a brief, tutorial overview of the product, then walks the 
reviewers through the materials step-by-step. An informal 
atmosphere and a free interchange of questions and answers among 
participants fosters the learning process. 


Inspections, on the other hand, are designed to uncover errors as 
early as possible and to ensure a high-quality product. The 
inspection team is a small group of peers who are technically 
competent and familiar with the application, language, and standards 
used on the project. The products to be reviewed (e.g., 
requirements, design diagrams, or source code) are given to the 
inspection team several days before the meeting. Inspectors 
examine these materials closely, noting all errors or deviations from 
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standards, and they come to the review meeting prepared to itemize 
and discuss any problems. 


In both walk-throughs and inspections, a designated team member 
records the minutes of the review session, including issues raised, 
action items assigned, and completion schedules. Closure of these 
items is addressed in subsequent meetings. 


In the requirements definition phase, walk-throughs of the 
requirements and specifications are conducted to ensure that key 
interested parties provide input while requirements are in a formative 
stage. Participants include the members of the requirements 
definition team, representatives of systems that will interface with 
the software to be developed, and application specialists from the 
development team. 


Prototyping 


During the requirements definition phase, prototyping may be 
needed to help resolve requirements issues. For mission support 
systems, analysts use prototyping tools such as MathCAD to test the 
mathematical algorithms that will be included in the specifications. 
For performance requirements, platform-specific performance 
models or measurement/monitoring tools may be used. 


MEASURES 
Objective Measures 


Three progress measures are tracked during the requirements 
definition phase: 


¢ Staff hours — i.e., the cumulative effort hours of the project 
staff 

+ Number of requirements with completed specifications versus 
the total number of requirements 

+» Number of requirements defined versus the total number of 
estimated requirements 


The sources of these data and the frequency with which the data are 
collected and analyzed are shown in Table 3-1. 
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staff hours 


completed 
specifications 


defined 
requirements 


Table 3-1. Objective Measures Collected During the 
Requirements Definition Phase 


FREQUENCY 


Staff hours (total and Requirements Weekly/monthly 
by activity) definition team and 

managers 

(time accounting) 


Requirements status Managers Biweekly/biweekly 


(percentage of 
completed 
specifications; number 
of requirements defined) 


Estimates of total Managers Monthly/monthly 
requirements, total 

requirements definition 

effort, and schedule 


Evaluation Criteria 


Effort should be gauged against estimates based on historical data 
from past projects of a similar nature. Monitor staff hours 
separately for each major activity. If schedules are being met but 
hours are lower than expected, the team may not be working at the 
level of detail necessary to raise problems and issues. 


To judge progress following the SCR, track the number of 


requirements for which specifications have been written as a 
percentage of the total number of requirements. (‘Total require- 
ments” includes those for which a need has been identified, but for 
which details are still TBD.) 


Monitor requirements growth by tracking the number of 
requirements that have been defined against an estimated total for the 
project. If requirements stability is an issue, consider tracking the 
number of changes made to requirements as well. Excessive growth 
or change to specifications point to a need for greater management 
control or to the lack of a detailed system operations concept. 
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PRODUCTS 


The key products of the requirements definition phase are the system 
and operations concept (SOC) document and the requirements and 
specifications document. The content and form of these products 
are addressed in the following paragraphs. 


System and Operations Concept Document 


The SOC document lists the high-level requirements, defines the 
overall system architecture and its operational environment, and 
describes how the system will operate within this environment. The 
document provides a base from which developers can create the 
software structure and user interface. The format recommended for 
the document is shown in Figure 3-3. 


The SOC is not usually updated after publication. During the 
requirements analysis phase, developers refine the reuse proposal 
contained in the document and publish the resulting reuse plan in the 
requirements analysis report. Similarly, developers refine the 
operational scenarios and include them in the requirements analysis, 
preliminary design, and detailed design reports. Because these and 
other pieces of the SOC are reworked and included in subsequent 
development products, it may not be necessary to baseline or 
maintain the SOC itself. 


Requirements and Specifications Document 


This document is produced by the requirements definition team as 
the key product of the requirements definition phase. It is often 
published in multiple volumes: volume 1 defines the requirements, 
volume 2 contains the functional specifications, and volume 3 
provides mathematical specifications. The document is distributed 
prior to the SRR, updated following the review to incorporate 
approved review items, and then baselined. 


The requirements and specifications document contains a complete 
list of all requirements — including low-level, derived requirements 
— and provides the criteria against which the software system will 
be acceptance tested. The functional or object specifications, which 
identify the input data, output data, and processing required to 
transform input to output for each process, provide the basis for 
detailed design and system testing. The document also includes the 
mathematical background information necessary to evaluate 
specified algorithms and to design the system correctly. 
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The recommended outline for the requirements and specifications document is presented in 
Figure 3-4. 


SYSTEM AND OPERATIONS CONCEPT DOCUMENT 


This document provides a top-down view of the system from the user’s perspective by describing 
the behavior of the system in terms of operational methods and scenarios. Analysts should 
provide the document to the development team by the end of the requirements definition phase. 
The suggested contents are as follows: 


1. Introduction 
a. Purpose and background of the system 


b. Document organization 


System overview 
a. Overall system concept 

b. System overview with high-level diagrams showing external interfaces and data flow 
c. Discussion and diagrams showing an ideal, high-level architecture for the system 


Reuse proposal 

a. Summary of domain and reuse analysis performed 

b. Description of potential candidates for reuse — architectural components, 
designs, operational processes, and test approaches — and associated trade-offs 

c. Discussion and diagrams of the proposed high-level architecture, as adjusted to 

incorporate reusable elements 


4. Operational environment — description and high-level diagrams of the environment in 
which the system will be operated 

a. Overview of operating scenarios . 

b. Description and high-level diagrams of the system configuration (hardware and software} 

c. Description of the responsibilities of the operations personnel 


5. Operational modes 

a. Discussion of the system's modes of operation (e.g., critical versus normal and 
launch/early mission versus on-orbit operations) 

b. Volume and frequency of data to be processed in each mode 

c. Order, frequency, and type (e.g., batch or interactive} of operations in each mode 


Operational description of each major function or object in the system 

a. Description and high-level diagrams of each major operational scenario showing all input, 
output, and critical control sequences 

b. Description of the input data, including the format and limitations of the input. Sample 

screens (i.e., displays, menus, popup windows) depicting the state of the function 

before receiving the input data should also be included. 

Process — high-level description of how this function will work 

Description of the output data, including the format and limitations of the output. 

Samples (i.e., displays, reports, screens, plots) showing the results after processing 

the input should also be included. _. i 

e. Description of status and prompt messages needed during processing, including 

guidelines for user responses to any critical messages 


ap 


7. Requirements traceability matrix mapping each operational scenario to requirements 


Figure 3-3. SOC Document Contents 
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REQUIREMENTS AND SPECIFICATIONS DOCUMENT 


This document, which contains a complete description of the requirements for the software 
system, is the primary product of the requirements definition phase. In the flight dynamics 
environment, it is usually published in three volumes: volume 1 lists the requirements, volume 2 
contains the functional specifications, and volume 3 provides the mathematical specifications. 


1. Introduction 
a. Purpose and background of the project 
b. Document organization 


System overview 

. Overall system concept 

. Expected operational environment (hardware, peripherals, etc.) 

. High-level diagrams of the system showing the external interfaces and data flows 
. Overview of high-level requirements 


Requirements — functional, operational (interface, resource, performance, reliability, safety, 
security), and data requirements 
a. Numbered list of high-level requirements with their respective derived requirements 
(derived requirements are not explicitly called out in source documents such as the SIRD 
or SORD, but represent constraints, limitations, or implications that must be satisfied to 
achieve the explicitly stated requirements) 
. For each requirement: 
(1) Requirement number and name 
(2) Description of the requirement 
(3) Reference source for the requirement, distinguishing derived from explicit 
requirements 
(4) Interfaces to other major functions or external entities 
(5) Performance specifications — frequency, response time, accuracy, etc. 


Specifications 
a. Discussion and diagrams showing the functional or object hierarchy of the system 
b. Description and data flow/object diagrams of the basic processes in each major 
subsystem 
c. Description of general conventions used (mathematical symbols, units of measure, etc.) 
d. Description of each basic function/object, e.g.: 
(1) Function number and name 
(2) Input 
(3) Process — detailed description of what the function should do 
(4) Output 
(5) Identification of candidate reusable software 
(6) Acceptance criteria for verifying satisfaction of related requirements 
7) Data dictionary — indicating name of item, definition, structural composition of the 
item, item range, item type 


Mapping of specifications to requirements — also distinguishes project-unique 
requirements from standard requirements for the project type (AGSS, dynamics 
simulator, etc.) 


Mathematical specifications — formulas and algorithm descriptions to be used in 
implementing the computational functions of the system 

a. Overview of each major algorithm 

b. Detailed formulas for each major algorithm 


Figure 3-4. Requirements and Specifications Document Contents 
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REVIEWS 


Two key reviews are conducted during the requirements definition 
phase: the system concept review and the system requirements 
review. The purpose, participants, scheduling, content, and format 
of these reviews are discussed in the subsections that follow. 


System Concept Review 


The SCR gives users, customer representatives, and other interested 
parties an opportunity to examine and influence the proposed system 
architecture and operations concept before detailed requirements are 
written. It is held during the requirements definition phase after 
system and operations concepts have been defined. In the flight 
dynamics environment, a full SCR is conducted for large, mission 
support systems. For smaller development efforts without complex 
extemal interfaces, SCR material is presented informally. The SCR 
format is given in Figure 3-5. 


SCR FORMAT 


Presenters — requirements definition team 


Participants 

+ Customer representatives 

« User representatives 

+ Representatives of systems and groups that will interface with the 
system to be developed 

* Senior development team representatives (application specialists) 

* Quality assurance (QA) representatives 

« System capacity/performance analysts 


Schedule — after the system and operations document is complete 
and before requirements definition begins 


Agenda — summary of high-level requirements (e.g., from SIRD and 
SORD) and presentation of system and operations concepts; 
interactive participation and discussion should be encouraged. 


Materials Distribution 

* The system and operations concept document is distributed 1 to 2 
weeks before the SCR. 

* Hardcopy material is distributed a minimum of 3 days before SCR. 


Figure 3-5. SCR Format 


SCR Hardcopy Material 
The hardcopy materials distributed for use at the review should 


correspond to the presentation viewgraphs. A suggested outline for 
the contents of SCR hardcopy material is presented in Figure 3-6. 
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HARDCOPY MATERIAL FOR THE SCR 


Agenda — outline of review material 
Introduction — purpose of system and background of the project 


High-level requirements 

a. Derivation of high-level requirements — identification of input (such as the SIRD and 
SORD) from project office, support organization, and system engineering organization 

b. Summary of high-level requirements 


System concept 

a. Assumptions 

b. Overall system concept 

c. List of major system capabilities 


Reuse considerations 

a. Existing systems reviewed for possible reuse 
b. Reuse trade-offs analyzed 

c. Proposed reuse candidates 


High-level system architecture 

a. Description and high-level diagrams of the proposed system architecture (hardware and 
software), including external interfaces and data flow 

b. Diagrams showing the high-level functions of the system — their hierarchy and 
interaction 


System environment 

a. Computers and peripherals 

b. Communications 

c. Operating system limitations and other constraints 


Operations concept 
a. Assumptions 
b. Organizations that provide system and support input and receive system output 
c. System modes of operation (e.g., critical versus normal and launch versus on-orbit 
operations) 
. Order, frequency, and type (e.g., batch or interactive) of operations in each mode 
. Discussion and high-level diagrams of major operational scenarios 
Performance implications 


Issues, TBD items, and problems — outstanding issues and TBDs and a course of action to 
handle them 


Figure 3-6. SCR Hardcopy Material Contents 


System Requirements Review (SRR) 


When the requirements and specifications document is distributed, the requirements 
definition team conducts an SRR to present the requirements, obtain feedback, and facilitate 
resolution of outstanding issues. The SRR format, schedule, and participants are given in 
Figure 3-7. 
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SRR FORMAT 


Presenters — requirements definition team 


Participants 
+ Customer representatives 

« User representatives 

e Configuration Control Board (CCB) 

» Senior development team representatives 
« System capacity/performance analysts 

+ Quality assurance representatives 


Schedule — after requirements definition is complete and before the 
requirements analysis phase begins 


Agenda — selective presentation of system requirements, 
highlighting operations concepts and critical issues (e.g, TBD 


requirements) 


Materials Distribution 

+ The requirements and specifications document is distributed 1 to 2 
weeks before SRR. 

» Hardcopy material is distributed a minimum of 3 days before SRR. 


TAILORING NOTE 


For very large or complex 
projects, hold a preliminary SRR 
to obtain interim feedback from 
users and customers. The 
format of the PSRR is the same 
as the SRR. Hardcopy material 
contains preliminary results and 
is adjusted to reflect work 
accomplished to date. 


The Configuration Control Board (CCB/is 
a NASA/GSFC committee that reviews, 
controls, and approves FDD systems, 


internal interfaces, and system changes. 
Among its duties are the approval of 
system baseline reviews (e.g., SRR, PDR) 
and baseline documents (e.g., require- 
ments and specifications, detailed design 
document). 


Figure 3-7. SRR Format 


SRR Hardcopy Material 


Much of the hardcopy material for the review 
can be extracted from the requirements and 
specifications document. An outline and 
suggested contents of the SRR hardcopy 
material are presented in Figure 3-8. 
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HARDCOPY MATERIAL FOR THE SRR 


Agenda — outline of review material 
Introduction — purpose of system and background of the project 


Requirements summary — review of top-level (basic) requirements developed to form 
the specifications 
a. Background of requirements — overview of project characteristics and major events 
b. Derivation of requirements — identification of input from project office, support 
organization, and system engineering organization used to formulate the requirements: 
e.g., the SIRD, SORD, memoranda of information (MOls), and memoranda of 
understanding (MOUs) 
. Relationship of requirements to level of support provided — typical support, critical 
support, and special or contingency support 
. Organizations that provide system and support input and receive system output 
. Data availability — frequency, volume, and format 
Facilities — target computing hardware and environment characteristics 
. Requirements for computer storage, failure/recovery, operator interaction, diagnostic 
output, security, reliability, and safety 
. Requirements for support and test software — data simulators, test programs, and 
utilities 
Overview of the requirements and specifications document — its evolution, 
including draft dates and reviews and outline of contents 


Interface requirements — summary of human, special-purpose hardware, and 
automated system interfaces, including references to interface agreement documents 
{IADs) and interface control documents (ICDs) 


Performance requirements — system processing speed, system response time, system 
failure recovery time, and output data availability 


Environmental considerations — special computing capabilities, e.g., graphics, 
operating system limitations, computer facility operating procedures and policies, support 
software limitations, database constraints, and resource limitations 


Derived system requirements — list of those requirements not explicitly called out in 
the SIRD, SORD, MOls, and MOUs, but representing constraints, limitations, or 
implications that must be satisfied to achieve the explicitly stated requirements 


Operations concepts 

a. High-level diagrams of operating scenarios showing intended system behavior from the 
user's viewpoint 

b. Sample input screens and menus; sample output displays, reports, and plots; critical 
control sequences 


Requirements management approach 

a. Description of controlled documents, including scheduled updates 
b. Specifications/requirements change-contro!l procedures 

c. System enhancement/maintenance procedures 


Personnel organization and interfaces 
Milestones and suggested development schedule 


Issues, TBD items, and problems — a characterization of all outstanding requirements 
issues and TBDs, an assessment of their risks (including the effect on progress), and a 
course of action to manage them, including required effort, schedule, and cost 


Figure 3-8. SRR Hardcopy Material Contents 
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EXIT CRITERIA 


Following the SRR, the requirements definition team analyzes all 
RIDs, determines whether requirements changes are necessary, and 
revises the requirements and specifications document accordingly. 
The updated document is sent to the configuration control board 
(CCB) for approval. Once approved, it becomes a controlled 
document — the requirements baseline. 


Use the following questions to determine whether the requirements 
and specifications are ready to be given to the development team for 
analysis: 


e Do specifications exist for all requirements for which 
information is available? Have TBD requirements been 
minimized? 


e Have external interfaces been adequately defined? 


¢ Are the specifications consistent in notation, terminology, and 
level of functionality, and are the algorithms compatible? 


* Are the requirements testable? 


e Have key exit criteria been met? That is, has the requirements 
and specifications document been distributed, has the SRR been 
successfully completed, and have all SRR RIDs been answered? 


When the answer to these questions is "yes," the requirements 
definition phase is complete. 


During and following formal reviews, 
review item disposition forms (RIDs) 
are submitted by participants to 
identify issues requiring a written 


response or further action. Managers 
are responsible for ensuring that all 
RIDs are logged and answered and 
resulting action items are assigned and 
completed. 
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SECTION 4 


THE REQUIREMENTS ANALYSIS PHASE 


Ree ESOT 


ENTRY CRITERIA 
+ System and operations concept completed 


« SRR completed 
« Requirements and specifications baselined 


PRODUCTS 


« Requirements analysis report 
* Software development/management plan 
« Updated requirements and specifications 


pe rez 


MEASURES 


Staff hours 

¢ TBD requirements 

e Requirements questions/answers 

¢ Requirements changes 

+ Estimates of system size, effort, 
and schedule 


e Requirements walk-throughs * 

e Requirements classification * 

« Requirements forms 

+ Requirements analysis methods and 
CASE tools 

* Prototyping 

* Project library 


>. mS 


EXIT CRITERIA 


* Requirements analysis report completed 

+ Software specification review (SSR) | 
completed 

« SSR RIDs resolved 


SANS 


KEY ACTIVITIES 


Requirements Definition Team 
« Resolve ambiguities, discrepancies, 
and TBDs in the specifications 
* Participate in the SSR 


Development Team 
« Analyze and classify requirements 
« Refine the reuse proposal 
. Identify technical risks 
+ Prepare the requirements analysis report 
e Conduct the SSR 


Management Team 

+ Prepare the software development/ 
management plan 

¢ Staff and train the development team 

» Interact with analysts and customers to X 
facilitate resolution of requirements issues § 

« Review the products of the requirements 
analysis process 

«Plan the transition to preliminary design 
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OVERVIEW 


The purpose of the requirements analysis phase is to ensure that the 
requirements and specifications are feasible, complete, and 
consistent, and that they are understood by the development team. 


Requirements analysis begins after the 
requirements definition team completes the 
requirements and specifications and holds the 
SRR. During requirements analysis, members of 
the development team carefully study the 
requirements and specifications document. They 
itemize and categorize each statement in the 
document to uncover omissions, contradictions, 
TBDs, and specifications that need clarification or 
amplification. The development team takes the 
analysis that was performed in the requirements 
definition phase to the next level of detail, using 
the appropriate analysis methodology for the 
project (e.g., structured or object-oriented 
analysis). When analysis is complete, the team 
presents its findings at an SSR. 


The development team works closely with the 
requirements definition team during the entire 
phase. The requirements definition team 
participates in walk-throughs, answers questions, 
resolves requirements issues, and attends the 
SSR. Meanwhile, the project manager plans the 
approaches to be used in developing the software 
system and in managing the development effort, 
obtains and trains the necessary staff, and reviews 
the products produced during the phase. 


Figure 4-1 is a high-level data flow diagram of the 
requirements analysis process, 
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TAILORING NOTE 


On large projects, requirements 
analysis begins at the PSRR. 
Key members of the develop- 


ment team examine the review 
materials, participate in the 
review itself, and begin 
classifying requirements 
shortly thereafter. 


Atypical development team comprises 

+ the project manager, who manages 
project resources, monitors progress, 
and serves as a technical consultant 

- the project (or task) leader, who 
provides technical direction and daily 
supervision 

+ the programmers and application 
specialists who perform the technical 
work 

+a quality assurance representative 

* a project librarian (see METHODS & 

TOOLS) 
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MILESTONES AND 
SCHEDULE INFO 


SOFTWARE DEVELOPMENT, 
MANAGEMENT PLAN 


CUSTOMER 


NOTE: The methodologies used in the requirements classification and analysis activities (processes 4.1 and 4.2 in 
the above DFD) are described under METHODS AND TOOLS below. The requirements analysis report (process 4.3) 
is discussed under PRODUCTS, and a separate subsection is devoted to the SSR (process 4.4). The planning activity 
(process 4.5) is outlined under MANAGEMENT ACTIVITIES and is described in detail in Section 3 of Reference 12. 


Figure 4-1. Analyzing Requirements 
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KEY ACTIVITIES 


In the requirements analysis phase, activities are divided primarily 
among the requirements definition team, the development team, and 
software development managers. The key activities that each 
performs during the requirements analysis phase are itemized in the 


x 
following subsections. Figure 4-2 is a sample timeline showing 
how these activities are typically scheduled. = 
= 
Activities of the Requirements Definition Team 
* Resolve ambiguities, discrepancies, and TBDs in the s 
specifications. Conduct the initial walk-throughs of the 
requirements and specifications for the development team and z 
participate in later walk-throughs. Respond to all developer = 
questions. . 
Resolve the requirements issues raised by the development team. = 
Incorporate approved modifications into the requirements and z 
specifications document. E 
* Participate in the SSR. q 
Activities of the Development Team E 
= 
° Analyze and classify requirements. Meet Seo METHODS AND i 
with the requirements definition team to TOOLS below for more E 
walk through and clarify each requirement 7 iniceroatlen about z 
and specification. Identify requirements and p at 5 
specifications that are missing, conflicting, classifications, and as 
ambiguous, or infeasible. Assign a analysis- methódologies. = 
classification of mandatory, requires 7 
review, needs Clarification, information = 
only, or TBD to each item in the = 
requirements and specifications document. = 
Use structured or object-oriented analysis to verify the 
specifications. Expand the high-level diagrams in the < 
requirements and specifications document to a lower level of 
detail, and supply missing diagrams so that all specifications are 
represented at the same level. Ensure that user interactions and = 
major data stores (e.g., attitude history files) are completely = 
specified. 
7 
Determine the feasibility of computer capacity and performance E 


requirements in view of available resources. Establish initial 
performance estimates by comparing specified 
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The contents of the 
requirements analysis 
report and the software 
development/management 


plan are covered under 
PRODUCTS below. The 


functions/algorithms with those of existing 
systems. Use the estimates to model overall 
performance (CPU, I/O, etc.) for the 
operational scenarios described in the SOC. 
Adjust the SOC scenarios to take these results 
into account. 


SSR is discussed separately 


at the endiof this section: Walk through the results of classification and 


analysis with the requirements definition team. 


* Refine the reuse proposal into a realistic plan. 
Analyze the software reuse proposal in the SOC 
in light of the existing software's current 
operational capabilities and any changes to the 
requirements baseline. 


REFERENCE 
See the Manager's 
Handbook for Software 
l Development and the * Identify areas of technical risk. Plan and 


h to Software Cost 
Pa area be edie conduct prototyping efforts or other appropriate 
techniques to minimize these risks. 


and 18, respectively) for 
guidance in estimating 

project size, costs , and 
schedule. * Prepare the requirements analysis report and 


distribute it before the SSR. 


* Conduct the SSR and resolve all RIDs. 


Activities of the Management Team 


* Prepare the software development/management plan 
(SDMP). Review the histories of related, past projects for 
applicable size, cost, and schedule data as well as lessons 
learned. Determine what resources are needed, develop a 
staffing profile, and estimate project costs. Identify project risks 
and plan to minimize them. Document the technical and 
management approaches that will be used on the project. 


e Staff and train the development team. Bring staff onto the 
project as soon as possible following SRR (or, on large 
projects, PSRR). Ensure communications among development 
team members, managers, and the requirements definition team. 


Also make certain that the requirements definition team is 
adequately staffed following SRR, so that TBD and changing 
requirements can be given prompt and thorough analysis. 


* Interact with analysts and customers to facilitate resolution 
of requirements issues. Work with team leaders to assess the 
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Conduct requirements walk-throughs Participate in walk-throughs 


Answer developer questions 
REQUIREMENTS 


DEFINITION . 
TEAM Incorporate changes to requirements 


y 
Participate in SSR 


E g 


Participate in walk-throughs 


Submit questions 


oS eZ. 


Classify requirements 


SOFTWARE Generate DFDs/OO diagrams Conduct analysis walk-throughs 
DEVELOPMENT 
TEAM Identify risks; conduct prototyping efforts; refine operational scenarios 


Refine the reuse proposal 
Produce the requirements analysis report 


y 


MANAGEMENT | Estimate resources and schedules; staff the development team 
TEAM 
Facilitate resolution of requirements issues; review products 


Prepare the SDMP 7 
Direct the SSR 


Plan the transition to preliminary design 


TME 


-Figure 4-2: Timeline of Key Activities in the Requirements Analysis Phase 


feasibility of proposed requirements changes and to estimate their impact on costs and 
schedules. 


Review the products of the requirements analysis process. Look at requirements 
classifications, data-flow or object-oriented diagrams, the data dictionary, the 
requirements analysis report, and the SSR hardcopy materials. Schedule the SSR and 
ensure participation from the appropriate groups. 


Plan an orderly transition to the preliminary design phase. Convey to the 
development team members the parts of the software development plan that apply to 
preliminary design (e.g., design standards and configuration management procedures) 
and instruct them in the specific software engineering approach to use during design. 
While the key team members are preparing for SSR, have the remainder of the 
development team begin preliminary design activities. 
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METHODS AND TOOLS 


The following methods, techniques, and tools are used to support 
the activities of the requirements analysis phase: 


¢ Requirements walk-throughs 

« Requirements classifications 

¢ Requirements forms 

* Structured and object-oriented requirements analysis 
e CASE tools 

* Prototyping 

+ The project library 


Each is discussed below. 
Requirements Walk-Throughs 


At the beginning of the requirements analysis phase, developers 
meet informally with analysts of the requirements definition team to 
go through the requirements and specifications. During these initial 
walk-throughs, analysts discuss each of the specifications, explain 
why certain algorithms were selected over others, and give 
developers the opportunity to raise questions and issues. 


After developers have analyzed and classified the requirements and 
specifications, they conduct walk-throughs of their results for the 
requirements definition team. One walk-through should be held for 
each major function or object in the system. During these later 
walk-throughs, both teams review all problematic specification items 
and discuss any needed changes to the requirements and 
specifications document. 


To ensure that all problem areas and decisions are documented, one 
member of the development team should record the minutes of the 
walk-through meeting. Developers will need the minutes to fill out 
requirements question-and-answer forms for any issues that require 
confirmation, further analysis, or other action by the requirements 
definition team. 


Requirements Classification 


When the development team is thoroughly familiar with the 
requirements and specifications document, they take each passage 
(sentence or paragraph) in the requirements and specifications 
document and classify it as either mandatory, requires review, needs 
clarification, information only, or TBD. 
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¢ An item is mandatory if it is explicitly defined in project-level 
requirements documents such as the SIRD or SORD, or if it has 
been derived from analysis of the project-level requirements. If 
mandatory items are removed from the specifications, the system 
will fail to meet project-level requirements. 


¢ If (on the basis of project-level requirements and the system and 
operations concept) there is no apparent need for a particular 
requirement or specification, then that item requires review (i.e., 
further analysis by the requirements definition team). The item 
must be deleted from the specification (by means of a 
specification modification) or moved into the mandatory 
category before CDR. 


e An item needs clarification when it is ambiguous, appears 
infeasible, or contradicts one or more of the other requirements 
or specifications. 


e An item is labelled as information only 
if it contains no requirement or 
specification per se. Such an item may 
provide background information, 
helpful hints for the software developer, 


W the requirements and 
specifications are available in a 


etc. database, enter the classifications 
and supporting commentary into 
P bk ` p 9 the database online. Otherwise, 
* A requirement or specification item 1S summarize each requirement or 
TBD if (1) the item contains a statement specification item, create a list of 
H : : the summaries, and use the lists 
such as "the process is TBD at this to assign classifications. 


time," or (2) information associated 
with the item is missing or undefined. 


Requirements Forms 


During the requirements analysis and subsequent phases, question- 
and-answer forms are used to communicate and record requirements 
issues and clarifications. Specification modifications are used to 
document requirements changes. 


The development team uses question-and-answer forms to track 
questions submitted to the requirements definition team and to verify 
their assumptions about requirements. Managers of the 
requirements definition team use the forms to assign personnel and 
due dates for their team's response to developers. Responses to 
questions submitted on the forms must be in writing. 


The question-and-answer form cannot be used to authorize changes 
to requirements or specifications. If analysis of the submitted 


48 


question-and- 
answer forms 


A 


Ml MT T 


aa ARIRE o AAM AU aE TE 


A'i 


A N 


f 


f 


mtn) © 


f 


IME 


4 


Section 4 - Requirements Analysis 


specification 
modifications 


question or issue reveals that a requirements change is needed, 
members of the requirements definition team draft a specification 
modification. Proposed specification modifications must be 
approved by the managers of both the requirements definition and 
the development teams and by the CCB. The requirements 
definition team incorporates all approved specification modifications 
into the requirements and specifications document. 


Analysis Methods and CASE Tools 


The methods and tools applicable for requirements analysis are the 
same as those recommended for the requirements definition phase in 
Section 3. The development team will generally use the same 
method as was used by the requirements definition team to take the 
analysis down to a level of detail below that provided in the 
specifications. This allows the development team to verify the 
previous analysis and to fill in any gaps that may exist in the 
document. If CASE tools were used in the requirements definition 
phase to generate data flow or object diagrams, it is important to use 
the same tools in the requirements analysis phase. The value of a 
CASE tool as a productivity and communication aid is greatly 
reduced if developers must re-enter or reformat the diagrams for a 
different tool. 


If the requirements definition team has used functional 
decomposition for their analysis and the development team needs to 
generate an object-oriented design, then extra analysis steps are 
required. The development team must diagram the details of the 
specification at a low level, then use the diagrams to abstract back up 
to higher-level requirements. This allows the team to take a fresh, 
object-oriented look at the system architecture and to restructure it as 
needed. 


Prototyping 


During the requirements analysis phase, prototyping activities are 
usually conducted to reduce risk. If unfamiliar technology (€.g., 
hardware or new development language features) will be employed 
on the project, prototyping allows the development team to assess 
the feasibility of the technology early in the life cycle when changes 
are less costly to effect. If system performance or reliability is a 
major, unresolved issue, the team can prototype critical operations 
or algorithms. 


On projects where the requirements for the user interface must be 


prototyped — either because the interface is critical to system 
success or because users are uncertain of their needs — a tool that 
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allows the developer to set up screens and 
windows rapidly is often essential. With such 
a tool, the developer can give the user the "look Caution must be exercised to ensure 


ay" . j that any prototyping activity that is 
and feel" of a system without extensive Soneta a e asa hat a defined 


wi 


programming and can obtain early feedback. goal, and is not being used as a means 
. to circumvent standard development 
The tool should be able to generate menus, procedures. See PRODUCTS 
multiple screens, and windows and respond to Section 4 for additional guidance on 
input. One such tool that has been successfully how to plan a prototyping effort. 
used in the SEL is Dan Bricklin's Demo 


Program. 
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In each software development project, one team member is assigned 

the role of librarian. During a project, the librarian (sometimes the 
called the software configuration manager) maintains the project librarian 
library, which is a repository of all project information. The 

librarian also maintains configured software libraries and operates 

various software tools in support of project activities. 


MW 


vm ATI 


The librarian establishes the project library during the requirements 
analysis phase. In general, the project library contains any written 
material used or produced by the development team for the purpose 
of recording decisions or communicating information. It includes 
such items as the requirements and specifications document, 
requirements question-and-answer forms, approved specification 
modifications, and the requirements analysis summary report. 
Necessary management information, such as the software 
development/management plan, is also included. 
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MEASURES 


The following paragraphs describe the measures and evaluation 
criteria that managers can use to assess the development process 
during the requirements analysis phase. 


Objective Measures 


The progress and quality of requirements analysis are monitored by 
examining several objective measures: 


TH om Ml Tha MLS 


e Staff hours — actual, cumulative hours of staff effort, as a total 
and per activity 

¢ Requirements questions and answers — the number of question- 
and-answer forms submitted versus the number answered 
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Table 4-1. Objective Measures Collected During the Requirements Analysis Phase 


FREQUENCY erase COLLECTION 


SOURCE {COLLECT/ANALY ZE) 

Staff hours (total and Developers and Weekly/monthly 
by activity) managers 

(via Personnel 

Resources Forms 

(PRFs)) 
Requirements (changes Biweekly/biweekly 
and additions to Managers 
baseline) (via Development 

Status Forms (DSFs)) 
Requirements (TBD Biweekly/biweekly 
specifications) Managers 
Requirements Biweekly/biweekly 
(Questions/answers) Managers (via DSFs) 
Estimates of total Monthly/monthly 
SLOC, total effort, Managers (via Project i 
schedule, and reuse Estimates Forms 

(PEFs)) 


* TBD requirements — the number of 
requirements classified as TBD versus the 
total number of requirements 

e Requirements changes — the total 
cumulative number of requirements for 
which specification modifications have been 
approved 

+ Estimates of system size, reuse, effort, and 
schedule — the total estimated number of 
lines of code in the system; the estimated 
number of new, modified, and reused 
(verbatim) lines of code; the total estimated 
staff hours needed to develop the system, 
and estimated dates for the start and end of 
each phase of the life cycle. 


The SEL uses 3 hardcopy forms to 
collect metrics during the requirements 
analysis phase. The Personnel 
Resources Form is used by the 
development team to record weekly 
effort hours. The Project Estimates 
Form is used by managers to record 
their monthly size and effort estimates. 
The Development Status Form is used 
to record the number of requirements 
changes, and number of requirements 
questions vs. answers. See Reference 
19 for detailed information about SEL 
data collection forms and procedures. 


For each of these measures, Table 4-1 shows who provides the 
data, the frequency with which the data are collected and analyzed, 
and whether data collection is continued from the requirements 
definition phase or newly initiated. 
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Evaluation Criteria 


Staff hours are usually graphed against a profile of estimated staff 
effort that is generated by the software development manager for the 
SDMP (Figure 4-5). This early comparison of planned versus 
actual staffing is used to evaluate the viability of the plan. 


In the flight dynamics environment, hours that are lower than 
expected are a serious danger signal — even if schedules are being 
met — because they indicate the development team is understaffed. 
If too few developers perform requirements analysis, the team will 
not gain the depth of understanding necessary to surface 
requirements problems. These problems will show up later in the 
life cycle when they are far more costly to rectify. 


A growing gap between the number of questions submitted and the 
number of responses received or a large number of requirements 
changes may indicate problems with the clarity, correctness, or 
completeness of the requirements as presented in the requirements 
and specifications document. Data from similar past projects should 
be used to assess the meaning of the relative sizes of these numbers. 


Because unresolved TBD requirements can necessitate severe design 
changes later in the project, the number of TBD requirements is the 
most important measure to be examined during this phase. 
Categorize and track TBD requirements according to their severity. 
TBD requirements concerning external interfaces are the most 
critical, especially if they involve system input. TBDs affecting 
internal algorithms are generally not so serious. 


A TBD requirement is considered severe if it could affect the 
functional design of one or more subsystems or of the high-level 
data structures needed to support the data processing algorithms. 
Preliminary design should not proceed until all severe TBD 
requirements have been resolved. A TBD requirement is considered 
nominal if it affects a portion of a subsystem involving more than 
one component. Preliminary design can proceed unless large 
numbers of nominal TBD requirements exist in one functional area. 
An incidental TBD requirement is one that affects only the internals 
of one unit. Incidental TBD requirements 
should be resolved by the end of detailed 
design. 


For each TBD requirement, estimate the effect 
on system size, required effort, cost, and 
schedule. Often the information necessary to 
resolve a TBD requirement is not available 
until later, and design must begin to meet fixed 
deadlines. These estimates will help determine 
the uncertainty of the development schedule. 
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MORE MEASURES 


Consider tracking these 
4 additional measures of 
z “a progress during the require- 


ments analysis phase: 

- number of requirements 
classified vs. total 
requirements 

» number of requirements 

diagrams completed vs. 
estimated total diagrams 


staff hours 


requirements 
questions 
and changes 


TBD 
requirements 
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DETAILED BULD [BUILD SYSTEM [ACCEPT-| SYSTEM 
ANALYSIS | DESIGN | DESIGN ANCE DELIVERY 
TESTING 


SECOND REPLAN —jJy- e ACTUAL DATA 


FIRST REPLAN 


ORIGINAL PLAN 


FULL-TIME-EQUIVALENT (40-HOUR WORK WEEKS) OF STAFF 


Explanation: The original staffing plan was based on an underestimate of the system size. Toward the end of the design phase, 40% more 
eflort was required on a regular basis. This was one of many Indicators that the system had grown, and the project was repianned accordingly. 
However, effort continued to grow when the second plan called for it to level off and decline. When it was clear that still more staff would be 
required to maintain progress, an audit was conducted. The audit revealed that the project was plagued with an unusually large number of 
unresolved TBDs and requirements changes that were causing an excessive amount of rework and that — as part of the corrective action — 
another replan was necessary. 


Figure 4-3. Effort Data Example — ERBS AGSS 


The growth of system size estimates is another key indicator of 
project stability. Estimating the final size of the system is the first 
step in the procedure for determining costs, schedules, and staffing 
levels (Section 3 of Reference 12). Make the initial estimate by 
comparing current requirements with information from past projects 
within the application environment. Update and plot the estimate 
each month throughout the life cycle. 


system size 
estimates 


As the project matures, the degree of change in 
the estimates should stabilize. If requirements 
growth pushes the system size estimate beyond 
: expected limits, it may be necessary to 
HN is poe beh te oer implement stricter change control procedures or 

the amount of deviation can to obtain additional funding and revise project 


show the degree that the plans. See Section 6 of Reference 12 for 


development process or product is nae : 4 : . . 
varying from what was expected, or it additional guidance in evaluating size estimates. 
can indicate that the original plan was in 

error. If the plan was in error, then 

updated planning data (i.e., estimates) 

must be produced. 
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PRODUCTS 


The following key products are produced during this phase: 


0 


+ The requirements analysis report 


All us 


¢ The software development/management plan 
* Updated requirements and specifications document 
¢ Prototyping plans (as needed) = 
These products are addressed in the paragraphs that follow. ` 
The Requirements Analysis Report : 
The requirements analysis report establishes a basis for beginning e 
preliminary design and is, therefore, a key product of the = 
requirements analysis phase. This report includes the following: z 
* The updated reuse plan (The original reuse proposal was = 
developed during the requirements definition phase and recorded = 
in the systems and operations concept document. It is adjusted g 
to reflect approved requirements changes and the results of = 
analysis of the reusable software's current capabilities.) Da ę 
* Updates to operational scenarios (in view of prototyping results, _ 
performance analyses, requirements changes, and functional = 
reallocations) = 
* The DFDs or object-oriented diagrams generated to analyze and ‘ 
complete the specifications = 
° A summary of the results of requirements analysis, highlighting £ 
problematic and TBD requirements, system constraints, and = 
development assumptions l = 
* An analysis of the technical risks of the project, as well as cost T 
and schedule risks resulting from TBD requirements or other 
factors = 
Figure 4-4 presents the format and contents of the requirements = 
analysis report. B 


IA 


The Software Development/Management Plan 


The SDMP provides a detailed exposition of the specific technical 
and management approaches to be used on the project. In the ` 
SDMP, the development team manager discusses how the 
recommended approach will be tailored for the current project and 
provides the resource and schedule estimates that will serve as a 
baseline for comparisons with actual progress data. 
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REQUIREMENTS ANALYSIS REPORT 


This report is prepared by the development team at the conclusion of the requirements analysis 
phase. It summarizes the results of requirements analysis and establishes a basis for beginning 
preliminary design. The suggested contents are as follows: 


Introduction — purpose and background of the project, overall system concepts, and 
document overview 


Reuse proposal — key reuse candidates and overall architectural concept for the system 


Operations overview — updates to system and operations concepts resulting from work 
performed during the requirements analysis phase 
a. Updated operations scenarios 
b. Operational modes, including volume and frequency of data to be processed in each 
mode, order, and type of operations, etc. 
c. Updated descriptions of input, output, and messages 


Specification analysis 
a. Summary of classifications (mandatory, requires review, information only, needs 

clarification, or TBD) assigned to requirements and specifications 
Problematic specifications — identification and discussion of conflicting, ambiguous, 
infeasible, untestable, and TBD requirements and specifications 
Unresolved requirements/operations issues, including the dates by which resolutions 
are needed 
Analysis of mathematical algorithms 


System constraints 
a. Hardware availability — execution, storage, peripherals 
b. Operating system limitations 
c. Support software limitations 

Performance estimates and models 

Development assumptions 


Risks, to both costs and schedules. (These should include risks related to TBD or changing 
requirements, as well as technical risks.) 


Prototyping efforts needed to resolve technical risks, including the goals and completion 
criteria for each prototyping effort 


Data flow or object-oriented diagrams — results of all structured or object-oriented analysis 
of the requirements performed during the requirements analysis phase 


Data dictionary — for the updated processes, data flows, and objects shown in the 
diagrams 


Figure 4-4. Requirements Analysis Report Contents 


The manager prepares the SDMP during the requirements analysis phase and keeps it up to 
date throughout the development life cycle. Because of the primary importance of this 
plan, it is described in detail in the Manager's Handbook for Software Development 
(Reference 12). 
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The SDMP includes a software development approach; a description of risks and risk 
mitigation; an initial estimate of the system's size; and estimates of the schedule, staffing, 
resources, and cost of the project. Figure 4-5 outlines the contents of the SDMP. 


SOFTWARE DEVELOPMENT/MANAGEMENT PLAN 


in some sections of the plan, material (shown in italics) is to be regularly added during the life of 
the project. Other sections should be revised and reissued if circumstances require significant 
changes in approach. 


INTRODUCTION 


1.1 Purpose — brief statement of the project's purpose. 
12 Background — brief description that shows where the software products produced 
by the project fit into the overall system. 
13 Organization and Responsibilities 
1.3.1 Project Personnel — explanation and diagram of how the development team 
will organize activities and personnel to carry out the project: types and 
numbers of personnel assigned, reporting relationships, and team members' 
authorities and responsibilities (see Reference 12 for guidelines on team 
composition}. 
1.3.2 interfacing Groups — list of interfacing groups, points of contact, and 
group responsibilities. 


1. 


STATEMENT OF PROBLEM — brief elaboration of the key requirements, the steps 
(numbered) to be taken to accomplish the project, and the relation (if any) to other projects. 


TECHNICAL APPROACH 


Reuse Strategy — high-level description of the current plan for reusing software from 
~ existing systems. i 

Assumptions and Constraints — that govern the manner in which the work will be 
performed. 

Anticipated and Unresolved Problems — that may affect the work and the 

expected effect on each phase. 

Development Environment — development computer and programming languages. 
Activities, Tools, and Products — for each phase, a matrix showing (a) the major 
activities to be performed, (b) the development methodologies and tools to be 
applied, and {c} the products of the phase. Includes discussion of any unique 
approaches or activities. 

Build Strategy — which portions of the system will be implemented in which builds 
and the rationale for this. Determined during the preliminary design phase. Updated 
at the end of detailed design and after each build. 


Be & RS 


b 


Figure 4-5. SDMP Contents (1 of 2) 
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MANAGEMENT APPROACH 


41 
42 


Assumptions and Constraints — that affect the management approach, 

including project priorities. 

Resource Requirements — tabular lists of estimated levels of resources required, 
including estimates of system size (new and reused SLOC), staff effort (managerial, 
programmer, and support) by phase, training requirements, and computer resources. 
Includes estimation methods or rationale used. Updated estimates are added at the 
end of each phase. 

Milestones and Schedules — list of work to be done, who will do it, and when it 

will be completed. Includes development life cycle (phase start and finish dates); 
build/release dates; delivery dates of required external interfaces; schedule for 
integration of externally developed software and hardware; list of data, information, 
documents, software, hardware, and support to be supplied by externa! sources and 
delivery dates; list of data, information, documents, software, and support to be 
delivered to the customer and delivery dates; and schedule for reviews (internal 

and external). Updated schedules are added at the end of each phase. 

Measures — a table showing, by phase, which measures will be collected to capture 
project data for historical analysis and which will be used by management to 

monitor progress and product quality (see Reference 12). If standard measures will 
be collected, references to the relevant standards and procedures will suffice. 
Describes any measures or data collection methods unique to the project. 

Risk Management — statements of each technical and managerial risk or 

concern and how it is to be mitigated. Updated at the end of each phase to 
incorporate any new concerns. 


PRODUCT ASSURANCE 


5.1 
52 


Assumptions and Constraints — that affect the type and degree of quality 

control and configuration management to be used. 

Quality Assurance — table of the methods and standards that will be used to 

ensure the quality of the development process and products (by phase). Where 
these do not deviate from published methods and standards, the table should 
reference the appropriate documentation. Methods of ensuring or promoting quality 
that are innovative or unique to the project are described explicitly. Identifies the 
person(s) responsible for quality assurance on the project, and defines his/her 
functions and products by phase. 

Configuration Management — table showing products controlled, as well as 

tools and procedures used to ensure the integrity of the system configuration 

(when is the system under control, how are changes requested, who makes the 
changes, etc.). Unique procedures are discussed in detail. If standard configuration 
management practices are to be applied, references to the appropriate documents 
are sufficient. Identifies the person responsible for configuration management and 
describes this role. Updated before the beginning of each new phase with detailed 
configuration management procedures for the phase, including naming conventions, 
directory designations, reuse libraries, etc. 


APPENDIX: PROTOTYPING PLANS — collected plans for each prototyping effort to be 
conducted on the project. 


PLAN UPDATE HISTORY — /ead sheets from each update of the SDMP, indicating 
which sections were updated and when the update was made. 


Figure 4-5. SDMP Contents (2 of 2) 
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Updated Requirements and Specifications 


During this phase, the requirements definition team prepares updates 
to the requirements and specifications document on the basis of 
approved specification modifications. Additional specification 
modifications may be approved as a result of discussion at the SSR 
or the RID process. The requirements definition team must ensure 
that the updated requirements and specifications document is 
republished shortly after the SSR, so that it will be available to the 
developers as they generate the software design. 


Prototyping Plans 


Managing a prototype effort requires special vigilance. Progress is 
often difficult to predict and measure. A prototyping effort may 
continue indefinitely if no criteria are established for evaluating the 
prototype and judging completion. Writing a plan for each 
prototyping activity, no matter how brief, is vital to establishing 
control. 


The length of the plan and the time to prepare it should be 
proportional to the size and scope of the prototyping effort. A one- 
page plan may be all that is required for small prototyping efforts. A 
brief plan need not be published separately but may, instead, be 
incorporated into the SDMP (Figure 4-5). 


The following items should be included in the plan: 


e Objective of the prototype — its purpose and use 

* Statement of the work to be done and the products to be 
generated 

* Completion criteria 

e Assessment methods — who will evaluate the prototype and 
how it will be evaluated 

e Technical approach 

+ Resources required — effort and size estimates, staff, hardware, 
software, etc. 

e Schedule 


The SDMP should contain summaries of the detailed prototyping 
plans. Each summary should describe the general approach, discuss 
the items to be prototyped, include effort estimates, provide a 
schedule, and discuss the rationale for the schedule. 
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SOFTWARE SPECIFICATION REVIEW 


At the conclusion of requirements analysis, the development team 
holds an SSR. This is a high-level review conducted for project 
management and the end users of the system. The SSR format, 
schedule, and participants are itemized in Figure 4-6. . 


SSR FORMAT 
Presenters — software development team 


Participants 

» Requirements definition team 

* Customer interfaces for both the requirements definition 
and software development teams 

» User representatives 

» Representatives of interfacing systems 

* Quality assurance representatives for both teams 

. System capacity/performance analysts 

» CCB 


Schedule — after requirements analysis is complete and before the 
preliminary design phase is begun 


Agenda — selective presentation of the results of requirements 
analysis, directed primarily toward project management and the 
users of the system 


Materials Distribution 

* The requirements analysis report and software development/ 
management plan are distributed 1 to 2 weeks before SSR. 

* Hardcopy material is distributed a minimum of 3 days before SSR. 


Figure 4-6. SSR Format 


SSR Hardcopy Material 


The hardcopy materials for the review will contain some of the same 
information found in the requirements analysis report. Keep in 
mind that there is some flexibility in selecting the most appropriate 
information to include in the presentation. The contents suggested in 
Figure 4-7 are intended as a guideline. 
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HARDCOPY MATERIAL FOR THE SSR 


1. Agenda — outline of review material 


2. introduction — background of the project and purpose of system 


3. Analysis overview — analysis approach, degree of innovation required in analysis, special 
studies, and results 


A Revisions since SRR — changes to system and operations concepts, requirements, and 
specifications effected following the SRR 


5. Reusable software summary 
a. Key reuse candidates — identification of existing software components that satisfy 
specific system specifications exactly or that will satisfy the specifications 
after modification 
b. Overall architectural concept for the system 
c. Matrix of requirements to be fulfilled by reused components 


6 Requirements classification summary 
a. List of requirements and specifications with their assigned classifications 
(mandatory, requires review, needs clarification, information only, or TBD) 
b. Problematic specifications — identification and discussion of conflicting, ambiguous, 
infeasible, and untestable requirements and specifications 
& Unresolved requirements/operations issues, including the dates by which resolutions to 
TBDs are needed (NOTE: This is a key element of the SSR.) 


7. Functional or object-oriented specifications 
a. Object diagrams or high-level data flow diagrams showing input, transforming 
processes, and output 
b. Data set definitions for external interfaces to the system 


Performance model — key estimates and results of modeling system performance 


Development considerations 

a. System constraints — hardware availability, operating system limitations, and support 
software limitations 

b. Utility, support, and test programs — list of auxiliary software required to support 
development (e.g., data simulators, special test programs, software tools, etc.) 

c. Testing requirements 

d. Development assumptions 


10. Risks, both to costs and schedules — includes how risks are identified, their potential 
impact, and how they will be managed. Covers risks related to TBD or changing 
requirements as well as technical risks 


11. Summary of planned prototyping efforts needed to resolve technical risks, including 
the goals and schedule for each effort and a summary of any prototyping conducted to date 


12. Key contacts — leaders of technical teams, application specialists, and other key project 
members 


13. Milestones and schedules — includes size estimates, development life cycle (phase start 
and finish dates), schedule for reviews (internal and external), build/release requirements, 
delivery dates of required external interfaces, schedule for integration of externally 
developed software and hardware 


Figure 4-7. SSR Hardcopy Material 
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Section 4 - Requirements Analysis 


EXIT CRITERIA 


To determine whether the development team is ready to proceed with 
preliminary design, consider the following questions: 


¢ Have all TBD requirements been identified and their impacts 
assessed? 


* Are performance requirements (e.g., timing, memory, and 
accuracy) clear? Are the requirements feasible, given the 
environmental constraints? Are sufficient computer resources 
available? 


e Have the key exit criteria for the phase been met? That is, has 
the requirements analysis report been distributed, has the SSR 
been successfully completed, and have all SSR RIDs been 
answered? 


When these criteria have been met, the requirements analysis phase 
is complete. 
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SECTION 5 


THE PRELIMINARY DESIGN PHASE 


ENTRY CRITERIA 


« Requirements analysis report generated 
* SSR completed 
. sod RIDs answered 


MEASURES 
* Units identified/designed 


« Requirements Q&As, TBDs, and changes a 


° Staff hours 
« Estimates of system size, effort, 
schedule, and reuse 


A RE AA AE nea 


METHODS AND TOOLS 


* Functional decomposition and 
object-oriented design ** 

* Prologs and PDL 

« Software engineering notebooks 

* Design walk-throughs*** 

* Design inspections*** 

* Reuse verification 

. aes methods 


SRC eT RET TTT NLL OO 


With the Cleanroom methodos 


EXIT CRITERIA 


« Preliminary design report generated 
«PDR completed 
« PDR RIDs answered 


KEY ACTIVITIES 


;| Development Team* 


e Prepare preliminary design diagrams 

. Prepare prologs and PDL 
for high-level functions/objects 

« Document the design in the preliminary 
design report 

* Conduct the PDR 


_|Management Team 


« Reassess schedules, staffing, training, and 
other resources 

* Plan, coordinate, and control requirements 

“changes 

« Control the quality of the preliminary 
design process and products 

e Plan the transition to detailed design 


Requirements Definition Team 
« Resolve outstanding requirements issues 
» Participate in design walk-throughs and 
PDR 


* All test activities are conducted by a separate test team established at the beginning of 


preliminary design. During this phase, the test team develops a usage profile for statistical 
testing, derives a list of functions to be tested, and drafts a preliminary system test plan. 
** Box structures (References 20,21) are used to develop the design and examine alternatives. 
***Cleanroom relies upon design walk-throughs for understanding and upon inspections for ; 
ensuring quality products. Both methods have been used extensively on SEL-monitored projects § 
and are Akoy elements of the nets recommended apprpadii ES 
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Section 5 - Preliminary Design 


OVERVIEW 


The purpose of the preliminary design phase is to define the high- 
level software architecture that will best satisfy the requirements and 
specifications for the system. 


During preliminary design, the development team uses the updated 
requirements and specifications document to develop alternative 
designs and to select an optimum approach. The team partitions the 
system into major subsystems, specifies all system and subsystem 
interfaces, and documents the design using structure charts or 
annotated design diagrams. Developers use an iterative design 
process that proceeds somewhat differently, depending on whether a 
functional decomposition or object-oriented design approach is 
chosen. l 


Early in this phase, developers examine the software components 
that are candidates for reuse and verify their compatibility with 
overall system requirements and the emerging design. Prototyping 
activities begun during the requirements analysis phase may 
continue and new prototyping efforts may be initiated. Developers 
also define error-handling and recovery strategies, determine user 
inputs and displays, and update operational scenarios. 


During this phase, the development team continues to work closely 
with analysts of the requirements definition team to resolve 
requirements ambiguities and TBDs. To ensure that the emerging 
design meets the system's requirements, developers send formal 
requirements questions to analysts for clarification, conduct walk- 
throughs, and subject all design products to peer inspection. 


The preliminary design phase culminates in 
the preliminary design review (PDR), during 


which developers present the rationale for TAILORING NOTE 


selecting the high-level system design. The 
preliminary design report documents the 


On projects with a high degree 
of reuse, the preliminary and 
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initial system design and is distributed for detailed design phases may be 
review prior to the PDR. combined. In that case, both 


preliminary and detailed design 
activities are conducted, but 
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: Figure 5-1 presents a high-level data flow aaral pan oy ony ane and 
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Fn 


wl 


yor 


` 


Section 5 - Preliminary Design 


REQUIREMENTS ANALYSIS 
REPORT 


OFDS OR 
OBJECT 
DIAGRAMS 
PROPOSED 


SYSTEM 
ARCHITECTURE UPOATED REQUIREMENTS AND SPECIFICATIONS 


DOCUMENT 


REUSABLE SOFTWARE 


DESIGN DECISIONS 


DESIGN WALK-THROUGH 
INFO 


CONSTRUCT 
REQUIREMENTS DESIGN 


OERNTION DIAGRAMS 
TEAM 5.2 HIGH-LEVEL DESIGN 


DIAGRAMS 


N WALK-THROUGH 
INFO 


GENERATE 
PROLOGS AND 
POL PRELIMINARY DESIGN REPORT 


PREPARE 
PREUMINARY 
DESIGN REPORT 


SOFTWARE UBRARIES 54 


RIDS AND CONDUCT 
RESPONSES PREUMINARY 
DESIGN 
REVIEW 

5.5 


POR 
PRESENTATION 


POR 
PARTICIPA POR HARDCOPY MATERIALS 


NOTE: The processes labelled 5.1, 5.2, and 5.3 are described in the KEY ACTIVITIES subsection. Prologs and PDL (5.3) and 
design methodologies (5.2) are also discussed under METHODS AND TOOLS. The PDR is described under REVIEWS, 
and the preliminary design document is covered in PRODUCTS. 


Figure 5-1. Developing the Preliminary Design 
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KEY ACTIVITIES 


The following are the key activities of the development team, the 
management team, and the requirements definition team during the 
preliminary design phase. The development team performs the 


principal activities of the phase. The requirements definition team 
concentrates on resolving outstanding TBD requirements and 
providing support to the development team. The relationships 
among the major activities of these groups are shown in Figure 5-2. 


Activities of the Development Team 


66 


Prepare preliminary design diagrams. 


Using functional 


decomposition or object-oriented techniques, expand the 
preliminary software architecture that was proposed in earlier 
phases. Idealize the expanded architecture to minimize features 
that could make the software difficult to implement, test, 


maintain, or reuse. 


Evaluate design options. Weigh choices according to system 
priorities (e.g., optimized performance, ease of use, reuse 
considerations, reliability, or maintainability). Use prototyping 
and performance modeling to test alternatives, especially in risk 


areas. 


Examine the software components that 
are candidates for reuse. If system 
response requirements are especially 
stringent, model the performance of the 
reusable components. Update the reuse 
plan to reflect the results of these reuse 
verification activities. 


Generate high-level diagrams of the 
selected system design and walk the 
analysts of the requirements definition 
team through them. Use the high-level 
design diagrams to explain the system 
process flow from the analyst's 
perspective. Focus on system and 
subsystem interfaces. Explain 
refinements to the operations scenarios 


arising from analysis activities and 


include preliminary versions of user 
screen and report formats in the walk- 
through materials. 


TAILORING NOTE 


Design diagrams, prologs, 
and PDL are required for all 
systems, regardless of the 
design methodology applied. 
METHODS AND TOOLS 
discusses ways these items 
are represented. 


Reuse verification encompasses 
designs, documentation, and 
test plans and data as well as 
code. See METHODS AND 
TOOLS for more details. 
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Section 5 - Preliminary Design 


Answer developer questions; resolve requirements issues, TBDs 


REQUIREMENTS 
DEFINITION 


TEAM Participate in PDR 


Participate in design walk-throughs 


Develop idealized design 


Evaluate design alternatives; prototype risk areas 


Conduct reuse trade-off analyses 
Prepare preliminary design diagrams 


Refine operational scenarios 


SOFTWARE 
DEVELOPMENT Conduct design walk-throughs 
TEAM 
Prepare prologs, PDL 
Prepare preliminary design report 
— 7 
Conduct design inspections 
Prepare and conduct POR | 
Resolve PDR RIDs| 
fo ee A 
Record project history data; reassess schedules, staffing, resources 
Plan and control requirements changes; control quality 
MANAGEMENT Update SDMP estimates 
TEAM 


Plan the transition to detailed design 


Direct the PDR 

Vv 

SSR POR 
SS Te —_— 


Figure 5-2. Preliminary Design Phase Timeline 
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e Prepare prologs and PDL for the high-level functions/objects. 
For FORTRAN systems, prepare prologs and PDL to one level 
below the subsystem drivers. For Ada systems, prepare and 
compile specifications for the principal objects in the system and 
construct sufficient PDL to show the dependencies among 
packages and subprograms. (See Figure 5-4.) 


Provide completed design diagrams, unit 
prologs/package specifications, and PDL 
to other members of the development team 
for independent inspection and 
certification. 


Contents of the preliminary 
design report and PDR 


* Document the selected design in the materials are provided under 


preliminary design report. Include the Age the PRODUCTS and REVIEW 
reuse plan, alternative design decisions, headings, respectively, in this 
section. Inspection and 


and external interfaces. certification procedures are 
covered under METHODS 


* Conduct the PDR and resolve RIDs. AND TOOLS. 


Record design changes for use in the 
detailed design phase, but do not update 
the preliminary design report. 


Activities of the Management Team 


During preliminary design, the manager's 
focus changes from planning to control. The 
following are the major management activities 
of this phase: 


e Reassess schedules, staffing, training, beth able edit i 
and other resources. Begin the software ollected byahemansaoment 
development history by recording lessons team muoughieu the us of the 

i sees project. See Section 9 for an 
learned and project statistics from the Palina SDH contents. 


requirements analysis phase. Include 
percentages of project effort and schedule 
consumed, growth of system size 
estimates, and team composition. 


Ensure that the development team contains 

a mix of software engineers and personnel experienced in the 
application area. If the project is large, partition the development 
team into groups, usually by subsystem, and appoint group 
leaders. Adjust staffing levels to compensate for changes in 
requirements and staff attrition, and ensure the team obtains the 
training it needs to meet specific project demands. 
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* Plan, coordinate, and control requirements changes. Interface 
with analysts and the customer to facilitate resolution of 
requirements issues and TBDs. Monitor the number and 
severity of requirements questions submitted to analysts and the 
timeliness of responses. Ensure that analysts and the customer 
know the dates by which TBDs need to be resolved and 
understand the impact of changing or undetermined requirements 
items. 


Assess the technical impact and cost of each specification 
modification. Constrain modifications that necessitate extensive 
rework and non-critical enhancements. 


* Control the quality of the preliminary design process and its 
products during day-to-day management activities. Ensure that 
design walk-throughs, inspections, and reviews are scheduled 
and conducted. Attend walk-throughs and, optionally, 
inspections and oversee the reporting, tracking, and resolution 
of the design issues that arise. Make certain that all requisite 
software documentation is generated and review the preliminary 
design document. Ensure that the team adheres to project 
standards and configuration management procedures. 


* Plan an orderly transition to the detailed design phase. 
Consider the impacts of TBDs, specification modifications, and 
schedule or team adjustments. Revise project estimates of 
effort, duration, and size and update corresponding sections of 
the SDMP. Develop the project build strategy and prepare a 
preliminary build plan reflecting prototyping results, project 
risks, and remaining TBDs. 


Increase team size if necessary to begin detailed design and 
address the training needs of additional personnel. Oversee the 
establishment of online libraries to store unit prologs, PDL, and 
reused code. While project and group leaders prepare for PDR, 
start the rest of the team on detailed design activities. 


Control the PDR and ensure that all exit criteria have been met 
before declaring the phase complete. 


Activities of the Requirements Definition Team 
During the preliminary design phase, the requirements definition 


team provides support to software developers through the following 
activities: 
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* Continue to resolve requirements issues and TBDs. Clarify 
ambiguous, conflicting, or incomplete requirements. Provide 
prompt, written replies to developers’ requirements questions 
and discuss these responses with developers. Respond to 
changes in high-level system requirements, evaluate the impact 
of each change, and prepare specification modifications 
accordingly. 


e Participate in design walk-throughs and the PDR. 
Thoroughly analyze the proposed design. Work with 
developers to refine the operational scenarios and preliminary 
user interface. Follow up with developers to address issues 
raised during the walk-throughs. 


Review the preliminary design report and all supporting 
hardcopy materials before PDR. Pose questions and provide 
critiques of the initial, high-level system design during the 
review meeting, and use RIDs to document serious 
discrepancies. 


METHODS AND TOOLS 


The primary methods and tools used during the preliminary design 
phase are 


¢ Functional decomposition and object-oriented design 

e Prologs and PDL 

e Software engineering notebooks (SENs) 

¢ Design walk-throughs 

* Design inspections 

e Reuse verification 

e Analysis methods: prototyping, performance modeling, and 
code analysis 


Functional Decomposition and Object- 
Oriented Design 


Design technologies are methods by which 


software developers define the major : ; 
During the design and implemen- 


components of a software system, describe tation phases, the software 

the interrelationships among components, and development and Aaima 

create a foundation for implementation. Suaian and anner torme and 

Design diagrams, structure charts, and spenificence modifications to record 
: a and resolve requirements issues. 

documentation support these methods. See METHODS AND TOOLS, 


Through these tools, developers demonstrate Section 4, for more details. 
that a chosen design approach incorporates 
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REFERENCE 


` A thorough discussion of 


object-oriented analysis 
and design is provided in 
References 11, 16, and 17. 


Section 5 - Preliminary Design 


each capability and interface specified 
in the requirements and specifications 
document. The two principal design 
technologies used on SEL-monitored 


Structured design projects are functional decomposition 


principles 


(Reference 13) 


and object-oriented design (OOD). 


form the basis of : i ae 
the functional When using a functional decomposition 


Aromon N design method, developers identify the 


the SEL. 


major functions of a system and 
successively refine them into smaller 
and smaller functionally oriented 
components. High levels in the design 
define the algorithmic abstraction (the 
"what" of the process), and the lower 
levels provide primitive operations that 
implement the higher level actions. 


In the flight dynamics environment, functional decomposition is 
normally used for the development of FORTRAN systems. When 
using this design approach, functional baseline diagrams (tree 
charts) are generated during preliminary design for all components 
to two levels below the subsystem drivers (as shown in Figure 5-3). 
The remaining design diagrams (levels 3 to N) are completed during 
detailed design. Separate structure charts may augment the 
diagrams; alternatively, interface information may be added directly 
to the tree charts. 


REFERENCE 


Cohesion and coupling, 
indicators of software 
system strength, 


reliability, and 
maintainability, are 
discussed in Reference 13. 


The SEL also recommends that functionally 
oriented designs employ the principles of 
information hiding, data abstraction, loose 
coupling, and cohesion. Components below the 
heavy line in the functional decomposition 
hierarchy of Figure 5-3 denote low-level routines 
or utilities whose details are deferred to the detailed 
design phase. However, developers must still 
understand the total system architecture to produce 
a correct preliminary design. 


When using an object-oriented approach, designers 
identify the abstract objects and their attributes that 
model the real-world system, define operations on 
those objects, and establish the interfaces between 
them. By focusing primarily on the objects (the 
"things" of the system) rather than on the actions 
that affect those objects, object-oriented techniques 
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MAIN 
EXECUTIVE 


SUBSYSTEM 
LEVEL 


SUBSYSTEM 
LEVEL 1 


SUBSYSTEM 
LEVEL 2 


(LEVELS BELOW THIS LINE ARE NOT 
ADORESSED UNTIL DETAILED DESIGN) 


SUBSYSTEM 
LEVEL 3 


— — — 
Be te 


TREE CHARTS 


a PRELIMINARY DESIGN SUBSYSTEM 
c- ee DETAILED DESIGN LEVEL N 
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Figure 5-3. Extent of the Design Produced for FORTRAN Systems During the 
Preliminary and Detailed Design Phases 


x 


allow designers to map their solutions more directly to the problem. 


In the flight dynamics environment, Ada is typically the language involved when OOD 
techniques are chosen. In the preliminary design of Ada systems, all packages and 
subprograms that address elements of the problem domain are identified. This includes all 
high-level objects necessary to implement the system capabilities, the functions and 
procedures affecting these objects, externally visible data, and all interfaces and 
dependencies. i l 
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Section 5 - Preliminary Design 


Developers use object-oriented, stepwise refinement until all subsystems, all packages, and 
all visible subprograms within those packages have been identified. Design of package 
bodies (the hidden elements shaded in Figure 5-4) is reserved until the detailed design 
phase. A generalized preliminary design diagram appropriate for an Ada system is shown 
in Figure 5-4. 


MAIN 
SUB- 
PROGRAM 


GENERIC 
| PACKAGE 


NOTE: The shaded elements of 
this figure represent hidden 
portions that are not specified 
until the detailed design phase. 


SUB- 


PROGRAM i 
2 VISIBLE 


PART 


Figure 5-4. Level of Detail Produced for Ada Systems During Preliminary Design 
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Prologs and PDL 


Comparable to the blueprint in hardware 
systems, prologs and PDL communicate the 
concept of the design to the level of detail 
necessary for implementation. The prolog 
provides a textual explanation of the unit's 
purpose and variables; PDL provides a formal, 
algorithmic specification for the unit. By using 
prologs and PDL, the developer is able to 
communicate the exact intent of the design to 
reviewers and coders. 


The SEL views the use of prologs and PDL 
during design as beneficial and cost-effective. 
Regardless of the design methodology chosen, 
developers are required to generate unit prologs 
and high-level PDL (consisting of such items as 
call dependencies, major logic branches, and 


REFERENCE 


in References 22 (Ada 
conventions) and 23 
(FORTRAN conventions). 


SEL conventions for 
y prologs and PDL are found 


TAILORING NOTE 


For large Ada systems with broad, 
flat designs, creating PDL for 
externally visible elements of al! 
packages and all visible 
subprograms entails extensive 
effort. When this situation exists, 


software managers should adjust 
effort and schedule allocations to 
allow sufficient time to complete 
this work in preliminary design. 


error-handling strategies) to complete the 
preliminary design. 


For FORTRAN systems, prologs and PDL are produced to one 
level below the subsystem drivers. For object-oriented systems, 
package specifications (which serve as prologs in Ada systems) and 
high-level PDL are generated for all program elements depicted in 
the design diagrams (see Figure 5-4). To identify interface errors, 
Ada PDL is always compiled; developers use templates and a 
language-sensitive editor (LSE) to standardize PDL structure. - 


Software Engineering Notebooks 


A software engineering notebook (SEN) is a workbook (i.e., a file 
folder or special notebook) that facilitates quality assurance and 
configuration management by consolidating printed information 
pertinent to a software component. This information becomes more 


detailed as the life cycle progresses. When printed documentation is ~ 


combined with online source files in later phases, the SEN provides 
a baseline history of an element's evolution through the development 
process. 


During preliminary design, developers initiate SENs at a subsystem 
or logical function level. Into these SENs they collect notes 
documenting design decisions, design diagrams, structure charts, 
and signed design inspection checklists for the subsystem or 
function. SENs are usually maintained by developers until after unit 
testing, when they are turned over to the project librarian. 
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oy 


Design Walk-Throughs 


Developers conduct design walk-throughs to ensure that both the 
requirements definition and development teams understand the 
system design as it takes shape. Developers distribute materials 
prior to the walk-through to participants, who include other 
developers, analysts, user representatives, representatives of 
systems that will interface with the software under development, and 
managers. During the meeting, developers carefully explain how 
operational aspects of the system (processing sequences and 
interfaces, screen formats, and user inputs) are reflected in the 
emerging design. Participants comment on how completely and 
= accurately developers have interpreted the system requirements. A 
z recording secretary notes discrepancies, errors, and inconsistencies 

and records action items. If significant issues remain at the end of 
3 the walk-through, a follow-up session may be scheduled. 


| 


The initial walk-through typically presents the overall, system level 
2 approach and is later followed by walk-throughs of subsystems and 
k i their major parts. When a project is large and the development team 
has been partitioned into groups, it is important that the group leader 
of any subsystem that interfaces with the subsystem being presented 
attend the session. 


Design Inspections 


z Whereas design walk-throughs focus on 
explaining operational aspects of the system 


Design inspections are conducted during 
both the preliminary and detailed design . 
phases because different units are 
designed in each phase. If a unit design 
was inspacted and certified during the 


preliminary design phase, it is not re- 
inspected during the detailed design 
phase unless it has been revised. See 
METHODS AND TOOLS in Section 6 for a 
detailed description of design inspection 
procedures. 


design, design inspections are independent, in- 
depth, technical reviews of the design 
diagrams, prologs, and PDL that are 
performed by software team peers. 
Inspections are held as logically related parts 
of the system design become clear and 
complete; they are scheduled when unit 
prologs and PDL have been generated to the 
required level. Developers distribute these 
materials for study by the inspection team prior 
to holding a working meeting. 


An inspection team consists of three or more members of the 
development team who are versed in the project's standards, 
development language, and system requirements. One member of 
the team acts as the moderator for the meeting. 
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Inspection team participants certify that unit logic and interfaces 
accurately represent the system requirements and that developers 
have applied correct design principles. Reviewers document their 
assessment and note weaknesses or interface conflicts on design 
inspection checklists (Figure 6-3). The signed checklist also 
certifies that the unit follows prescribed project standards and 
conventions. 


Reuse Verification 


Reuse verification is the process of determining which of the 
existing software components specified in the reuse proposal should 
be integrated into the new system design. During reuse verification, 
developers examine code and documentation from sources such as 
the FDF's Reusable Software Library (RSL). Developers draw on 
their experience in considering the integrity of the overall system 
architecture, the clarity of the design, and system performance. 


When the reuse plan recommends using large portions of existing 
systems, developers must assess the trade-offs of compromising an 
optimum system design to make it compatible with existing 
software. They must consider long-term impacts on total software 
development costs, design clarity, performance requirements, 
system size, maintainability, and reliability when weighing design 
options. 


When factors (such as incompatible language versions or a high 
incidence of operating system-specific calls) prohibit an existing 
component from being reused, developers can study the software 
and its documentation to understand its function, organization, and 
data structures before designing a component compatible with the 
new system. Thus, reused experience is shared across projects even 
when explicit design and code cannot be. 


Analysis Methods: Prototyping, Performance Modeling, and 
Code Analysis 


During preliminary design, the development team uses prototyping 
to validate design concepts and to test the trade-offs of design 
options. Developers compose prototype drivers (or scaffolding 
code) to exercise components planned for reuse in the new system. 
They also prototype user screens and report formats. To confirm 
that existing components will meet the new system's performance 
requirements, developers couple prototyping with performance and 
source code analysis, as described below. 
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units 
designed 


Performance modeling is a means of predicting how efficiently a 
program will use resources on a given computer. To generate these 
predictions, developers use such parameters as the number of units 
anticipated, the volume and frequency of the data flow, memory 
usage estimates, and the amount of program I/O. Analogy with 
existing, similar systems may also be used. Results of performance 
modeling assist developers in deciding among design options. 


If executable code exists (e.g., scaffolding code or reused units), 
developers can run performance analyzers, such as Problem 
Program Evaluator (PPE) or the VAX Performance and Coverage 
Analyzer, concurrently with the code. These dynamic analyzers 
generate information such as CPU time, I/O counts, and page fault 
data that help developers identify areas of the code (e.g., inefficient 
loop structures or calculations) that need to be redesigned. 


Static code analyzers (e.g., WAX Source Code Analyzer) are tools 
that read the source code of existing components and generate 
information describing their control structure, symbol definitions, 
and variable occurrences. A developer can examine the call trees 
produced by a static code analyzer to assess the scope of a proposed 
change to a reusable component's interfaces. The designer can then 
decide which approach is better — to modify and test all affected 
units or to design a new component. 


MEASURES 


During preliminary design, managers continue to use the objective 
measures of the requirements analysis phase. They also begin to 
monitor additional progress data. The following measures are 
collected: 


* The number of units designed versus the number identified 
e Requirements questions and answers, TBDs, and changes 
* Staff hours 

¢ Estimates of system size, effort, schedule, and reuse 


The source of these data and the frequency of data collection and 
evaluation are shown in Table 5-1. 


Evaluation Criteria 


Project leaders estimate the number of units that will be needed to 
represent the preliminary system design. Against this estimate, they 
track the number of unit designs (prolog and PDL) that have been 
generated and the number certified. Management plots assist in 


77 


ection 5 - Preliminary Design 


Table 5-1. Objective Measures Collected During the Preliminary Design Phase 


Stalf hours (total and 
by activity) 


Requirements (changes 
and additions to 
baseline) 


Requirements (TBD 
specifications) 


[mene | ee 


Developers and 
managers 
(via PRFs} 


Managers 
(via Development 
Status Forms (DSFs)) 


Managers 
(via DSFs) 


FREQUENCY DATA COLLECTION 


(COLLECT/ANALYZE) CONTINUED | BEGUN 


Weekly/monthly 


Biweekly/biweekly 


Biweekly/biweekly 


Biweekly/biweekly 


Requirements Managers (via DSFs) 
(questions/answers) 


Estimates of total SLOC | Managers (via PEFs) Monthly/monthly 


(new, modified, reused), 
total units, total effort, 
and schedule 


Status (units planned/ Managers (via DSFs) Biweekly/biweekly 


designed certified) 


discovering trends or plateaus in this progress 
data that signal impending difficulties. The 
graph of units certified should closely follow 
and parallel the graph for units designed in a 
fairly smooth curve. Sharp increases (a "stair- 
step” graph) will appear when many units are 


* 


(total units) 


The number of units designed 


r 


mt 


AUN 


hurriedly certified together in an effort to meet should:not bo ushd-asa measu 
schedules. of unit completion. The correct 


completion measure is the number 


ml l 


of certified unit designs. 


During preliminary design, a widening gap 
between the number of requirements questions 
submitted and the number of answers received 
can be an early warning signal of impending 
rework and eventual schedule slippage. A 
high number of questions and answers when 
compared with systems of similar size and 
complexity is interpreted the same way. 


requirements 
questions 
and answers 


ARIN 


aR ni 


Tracking the number of TBD requirements that persist into the 
preliminary design phase, especially those concerning external 


: i ; A TBD requirements 
interfaces and hardware changes, is crucial because TBDs represent a 


hip tts Wty 
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requirements 
changes 


Numbers of Q&As, TBDs, and 
specification modifications can 
change rapidly. Plots of these 
factors help managers to assess 


project status and 
highlight problem questions 
areas. 


staff hours 


SEL managers update effort, schedule, 
and size estimates approximately once 
a month and organize these estimates 
on the Project Estimates Form (PEF). 


Data from these forms are used to 


generate system growth and progress 


plots. 


estimates 


Although a number of measures of 
design quality (strength, etc.) have 


been proposed, the SEL has not 


yet found objective measures that 
provide significant insight into the 
quality of a design. 
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incompleteness in the design and increase the 
potential for rework. TBDs should diminish 
quickly in the early life cycle phases. 


Specification modifications may be issued in 
response to development team questions or 
external project influences such as hardware 
changes. The number and severity of 
specification modifications resulting from 
developers’ questions reflect the quality of the 
requirements and specifications document. 
These changes are normally addressed by the 
development and requirements definition teams. 
The number and severity of specification 
modifications from external causes have more 
far-reaching implications and should alert 
managers to anticipate long-term perturbations in 
schedule and effort estimates. 


Significant deviations between planned and 
actual staff effort hours warrant close 
examination. When a high level of effort is 
required to meet schedules, the development 
team's productivity may be low or the problem 


_may be more complex than originally realized. 


Low staff hours may indicate delayed staffing on 
the project or insufficient requirements analysis. 
The effects of these last conditions are not 
always immediately obvious in preliminary 
design, but will surface dramatically as design 
deficiencies during subsequent phases. 


Managers can now use the number of units 
planned and average unit size to refine estimates 
of system size. The number of reused units in 
the system also becomes firmer; managers 
identify units to be reused without change versus 
those to be modified and revise effort and 
schedule projections accordingly. 


Estimates are a key management measure; widely 
varying estimates or sharp increases always 
warrant investigation. In the preliminary design 
phase, excessive growth in system size estimates 
is generally seen when requirements are unstable 
or change control mechanisms are ineffective. 
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Section 5 - Preliminary Design 


PRODUCTS 


The primary product of the preliminary design phase is the high- 
level design of the system, as documented in the preliminary design 
report. The report incorporates the results of reuse verification and 
prototyping and forms the basis for the detailed design document. 
Because of their size, unit prologs and PDL are normally published 
in a separate volume that is identified as an addendum to the report. 


An outline of the preliminary design report, showing format and 
content, is provided as Figure 5-5. 


PRELIMINARY DESIGN REVIEW 


The phase concludes with the preliminary design review (PDR), 
during which the development team presents the high-level system 
design and the rationale for choosing that design over alternatives. 
Also highlighted in the PDR are explanations of external system 
interfaces, revisions to operations scenarios, major TBDs for 
resolution, and issues affecting project quality and schedule. 


Materials presented at PDR do not necessarily convey the technical 
depth that the development team has achieved during preliminary 
design; details of this technical effort are documented in the 
preliminary design report. Developers limit the PDR presentation to 
the nominal operating scenarios and significant contingency cases. 


The presentation is directed to project managers, users, the 
requirements definition team, and the CCB. Applications 
specialists, analysts, quality assurance personnel, and managers 
perform a detailed technical review of the preliminary design 
material prior to attending the formal review. They evaluate the 
system design and provide comments and critiques to the 
development team during and immediately after the presentation. 
RIDs are used by participants to record any issues that still need to 
be resolved. The PDR format and schedule are shown in Figure 
5-6, followed by an outline of the PDR hardcopy materials (Figure 
5-7). 
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PRELIMINARY DESIGN REPORT 


This report is prepared by the development team as the primary product of the preliminary design 
phase. It presents the high-level design of the system and forms the basis for the detailed design 
document. The suggested contents are as follows: 


1. Introduction — purpose and background of the project, overall system concepts, and 
document overview : 


2. Design overview 
. Design drivers and their order of importance (e.g., performance, reliability, hardware, 
memory considerations, operating system limitations, language considerations, etc.) 
. Results of reuse tradeoff analyses; the reuse strategy 
. Critique of alternative designs 
. Discussion and high-level diagrams of the selected system design, showing hardware 
interfaces, external data interfaces, interconnections among subsystems, and data flow 
. Atraceability matrix of the subsystems against the requirements 
Design status 
List of constraints, concerns, and problem areas and their effects on the design 
List of assumptions and possible effects on design if they are wrong 
List of TBD requirements and an assessment of their effect on system size, required 
effort, cost, and schedule 
ICD status 
Status of prototyping efforts 
g. Development environment (i.e., hardware, peripheral devices, etc.) 


Operations overview 
. Operations scenarios/scripts (one for each major product that is generated). Includes the 
form and volume of the product and the frequency of generation. Panels and displays 
should be annotated to show what various selections will do and should be traced to a 
subsystem 
. System performance considerations 
Error recovery strategies (automatic fail-over, user intervention, etc.) 


Design description for each subsystem or major functional breakdown: 

. Discussion and high-level diagrams of subsystem, including interfaces, data flow, and 
communications for each processing mode 

. High-level description of input and output 
High-level description of processing keyed to operator-specified input and actions in 
terms of points of control, functions performed, and results obtained (both normal and 
abnormal, i.e., error processing and recovery} 

. Structure charts or object-oriented diagrams expanded to two levels below the 
subsystem driver 

. Prologs (specifying the unit's purpose, operation, calling sequence arguments, 
external references, etc.) and program design language (PDL) for each identified unit 
{Prologs and PDL are normally published in a separate volume.) 


Data interfaces for each internal and external interface: 
a. Description, including name, function, frequency, coordinates, units, and computer type, 
length, and representation 
b. Format 
(1) Organization, access method, and description of files (i.e., data files, tape, etc.) 
(2) Layout of frames, samples, records, and/or message blocks 
(3) Storage requirements 


Figure 5-5. Preliminary Design Report Contents 
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PDR FORMAT 


Presenters — software development team 


Participants 

« Requirements definition team 

* Quality assurance representatives from both teams 
» Customer interfaces for both teams 

« User representatives 

* Representatives of interfacing systems 

* System capacity/performance analysts 


. CCB 


Schedule — after the preliminary design is complete and 
before the detailed design phase begins 


Agenda — selective presentation of the preliminary design 
of the system 


Materials Distribution 

« The preliminary design report is distributed at least 1 week before 
PDR. 

e Hardcopy material is distributed a minimum of 3 days before PDR. 


Figure 5-6. PDR Format 


EXIT CRITERIA 


To determine whether the development team is ready to proceed with 
detailed design, managers should ask the following questions: 


* Have all components that are candidates for reuse been 
analyzed? Have the trade-offs between reuse and new 
development been carefully investigated? 


e Have developers evaluated alternate design approaches and 
chosen the optimum design? 


¢ Have all design diagrams, prologs and PDL (or package 
specifications, if applicable) been generated to the prescribed 
level? Have they been inspected and certified? 


¢ Have the key exit criteria been met? That is, has the preliminary 
design report been produced and distributed, has the PDR been 
successfully completed, and have all PDR RIDs been answered? 


When the manager can answer "yes" to each of these questions, the 
preliminary design phase is complete. 
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HARDCOPY MATERIAL FOR THE PDR 


Agenda — outline of review material 
Introduction — background of the project and system objectives 


Design overview 

. Design drivers and their order of importance (e.g., performance, reliability, hardware, 
memory considerations, programming language, etc.) 

. Results of reuse tradeoff analyses (at the level of subsystems and major components) 

. Changes to the reuse proposal since the SSR 

. Critique of design alternatives 

. Diagram of selected system design. Shows products generated, interconnections among 
subsystems, external interfaces. Differences between the system to be developed and 
existing, similar systems should be emphasized 
Mapping of external interfaces to ICDs and ICD status 


System operation 

a. Operations scenarios/scripts — one for each major product that is generated. Includes 
the form of the product and the frequency of generation. Panels and displays should be 
annotated to show what various selections will do and should be traced to a subsystem 

b. System performance considerations 

c. Error recovery strategies 


Major software components — one diagram per subsystem 
Requirements traceability matrix mapping requirements to subsystems 


Testing strategy 

a. How test data are to be obtained 

b. Drivers/simulators to be built 

c. Special considerations for Ada testing 


Design team assessment — technical risks and issues/problems internal to the software 
development effort; areas remaining to be prototyped 


Software development/management plan — brief overview of how the development 
effort is conducted and managed 


Software size estimates — one slide 
Milestones and schedules — one slide 


Issues, problems, TBD items beyond the control of the development team 
a. Review of TBDs from SSR 

b. Other issues 

c. Dates by which TBDs/issues must be resolved 


Figure 5-7. PDR Hardcopy Material 
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PRELUM}: | DETAILED fe: 
CYCLE NITION: ITS fo NARY. | DESIGN Ta 
PHASES i iS | DESIGN 


SECTION 6 


THE DETAILED DESIGN PHASE 


= oe PHASE HIGHLIG 
: nee Sa easier $ EEEE Be Ie 
ENTRY CRITERIA 
* Preliminary design report generated * Detailed design document generated 
* POR completed : * CDR completed* 
«= PDR RIDs answered 


KEY ACTIVITIES 


* Detailed design document 


* Build plan* * Prepare detailed design diagrams 
* Build/release test plan* e Conduct design wallerntolahs 
e Refine the operational scenarios 
* Complete prologs and PDL for all units 
+ Provide input to the build plan and begin 
the build/release test plan* 
+ Units designed/identified gy ceo he det ee desigidocument 
e Requirements Q&As, TBDs, and changes 
* Staff hours 
* Estimates of system size, effort, =] »° Assess lessons learned from the 
schedule, and reuse 7 preliminary design phase 
* CPU hours 3] e Control requirements changes 
A * Control the quality of the design process . 
e Prepare the build plan 
« Coordinate the transition to implementation 


- n 8 : Direct the CDR 
A METHODS AND TOOLS 2 


ances A Requirements Definition Team 
: * Functional decomposition and A + Resolve remaining requirements issues 
object-oriented design + Participate in design walk-throughs and 
» Reuse verification ; CDR 


* Analysis methods 2 
* Design walk-throughs =] Acceptance Test Team 


* Design inspections J > Begin work on the acceptance test plan 
* Prologs and PDL d « Prepare portions of analytical test plan 
needed fo 


RRR 


Sits 


RS R Adan Se ee AIARA 
*On projects using t eanroom methodology, each build consists of a detailed desig 
phase followed by an implementation phase. Statistically selected tests are planned 
and performed by a separate test team, which compiles, integrates, and tests each 


build as soon as it has passed coding inspection. An informal review is conducted 
after the design phase in each build. 
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OVERVIEW 


The purpose of the detailed design phase is to produce a completed 
design specification that will satisfy all requirements for the system 
and that can be directly implemented in code. 


Unless comments expressed at the PDR indicate serious problems or 
deficiencies with the preliminary design, detailed design begins 
following the PDR. The detailed design process is an extension of 
the activities begun during preliminary design. The development 
team elaborates the system architecture defined in the preliminary 
design to the unit level, producing a complete set of code-to 
specifications. The primary product of the phase is the detailed 
design document, which contains the design diagrams, prologs, and 
PDL for the software system. 


During this phase, the development team conducts walk-throughs of 
the design for the requirements definition team and subjects each 
design specification to peer inspection. At the conclusion of the 
phase, the completed design is formally reviewed at the CDR. 


Figure 6-1 shows the major processes of the detailed design phase. 


KEY ACTIVITIES 


While the development team is generating the detailed design, the 
requirements definition team continues to resolve the remaining 
requirements issues and TBDs. As soon as requirements questions 
and changes level off, an additional team is formed to prepare for 
acceptance testing. This acceptance test team usually consists of the 
analysts who will use the system, along with some of the staff who 
prepared the requirements and specifications document. 


The activities of the development team, the management team, the 
requirements definition team, and the acceptance test team are 
itemized below. A suggested timeline for the performance of these 
activities is given in Figure 6-2. 


Activities of the Development Team 


* Prepare detailed design diagrams to the lowest level of detail, 
i.e., the subroutine/subprogram level. Successively refine each 
subsystem until every component performs a single function and 
can be coded as a single unit. 
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een mM 
PREUMINARY DESIGN REPORT 


rae cee 


UPDATED REQUIREMENTS AND SPECIFICATIONS 
DOCUMENT 
CE 


REFINED 
OPERATIONAL 
SCENARIOS 
DETAILED 
DESIGN 
DIAGRAMS 
DESIGN WALK-THROUGH 
INFO 


REQUIREMENTS 
DEFINITION 
TEAM 


GENERATE 
PROLOGS AND PO 
(ALL UNITS) 
6.2 


PAOLOGS AND PDL 
aa 


DETAILED DESIGN DOCUMENT 
UNKS TO RSL 


INCORPORATE 
REUSABLE 


UNITS CONFIGURED SOFTWARE LIBRARIES 


63 
PREPARE 
DETAILED DESIG. 
MODIFIED PROLOGS AND POL DOCUMENT 


REUSABLE SOFTWARE LIBRARY 


PRESENTATION 


NOTE: The processes labelled 6.1, 6.2, and 6.3 are described in the 

KEY ACTIVITIES subsection. The detailed design document is DR 

covered under PRODUCTS. A separate subsection describes the PARTICIPAN TOR HARDCOPY MATERIALS 
format and contents of the CDR. 


Figure 6-1. Generating the Detailed Design 


* Conduct design walk-throughs. Walk through each major function or object with 
-= other developers and the requirements definition team. On small projects (e.g., 
simulators), conduct one walk-through per subsystem, On systems of 150 KSLOC or 
more (e.g., AGSSs), hold two walk-throughs per subsystem — one in the early stages 

of detailed design and one later in the phase. 


* Refine the operations scenarios for the system in light of the results of prototyping, 
performance modeling, and design activities. Finish specifying detailed input and 
output formats for each subsystem, including displays, printer and plotter output, and 
data stores. 
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Answer developer questions; resolve requirements issues, TBDs 


REQUIREMENTS | — — =- = — OT OT y 
DEFINITION Participate in design walk-throughs y 


TEAM . Participate in CDR 


Complete all prototyping 


Refine operational scenarios 


Finalize design diagrams 
eS a Gee aN ene ae ae eee Se 


Conduct design waik-throughs 


Prepare all prologs and PDL 


SOFTWARE op tr fet Nien gaan ade Sas cee eg eee aad Neal ec aap sy eee 
DEVELOPMENT Conduct design inspections 
TEAM ee, 
Prepare detailed design report 
eee eee 
NOTE: Dashed tines indicate Prepare build test plan 
that the activity is intermittent 
Prepare and conduct 
Resolve CDR RIDs 
ACCEPTANCE = 
TEST Begin to prepare the acceptance test plan 
TEAM 
Plan analytical tests for Build 1 
Record project history data, reassess schedules, staffing, resources 
Plan and control requirements changes; control quality 
MANAGEMENT Prepare build plan 
TEAM Update SDMP estimates 


Direct the CDR 


Coordinate the transition to implementation 


v 
PDR COR 
TIME H 


Figure 6-2. Timeline of Key Activities in the Detailed Design Phase 
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Complete all prototyping efforts. The detailed design presented 
at CDR should contain no uncertainties that could have been 
resolved through prototyping. 


* Complete prologs and PDL for all units. Generate prologs and 
PDL for the new units specified in the design diagrams. On Ada 
projects, package specifications and PDL should also be 
compiled. This means that package bodies are compiled, and 
that, at a minimum, subprogram bodies each contain a null 
Statement. 


Identify all units (from the RSL or other sources) that will be 
reused, either verbatim or with modifications. Transfer existing 
units that require modification into online libraries, and revise 
the prologs and PDL of these units as necessary. 


Ensure that all unit designs are formally inspected and certified 
(see Methods and Tools below). File the completed checklist in 
the SEN for the unit. 


* Provide input to the build plan and begin the build test plan. 
Provide the technical information needed for the build plan to the 
management team; include the order in which units should be 
implemented and integrated. Prepare the test plan for the first 
build and review it at CDR. (See Section 7 for the plan's format 
and contents.) 


* Prepare the detailed design document as a basis for the CDR. 
Ensure that the team librarian adds all documentation produced 
during the phase to the project library. (The only exceptions are 
SEN materials, which are maintained by individual developers 
until the unit has been coded and tested.) 


e Conduct the CDR and respond to all CDR RIDs. 


Activities of the Management Team 


With a few exceptions, the activities of the management team during 
the detailed design phase parallel those of the previous phase. 


* Assess lessons learned from the preliminary design phase 
and record information for the software development history, 
including schedule data and project statistics. Reassess 
schedule, staffing, and resources in view of these data. 


* Control requirements changes. Continue to monitor the 
number and scope of requirements questions and answers. 
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Ensure that analysts and the customer understand the potential 
impact of each requirement TBD and proposed specification 
modification. 


Control the quality of the detailed design process and its 
products. Ensure adherence to design standards, configuration 
management procedures (especially change control), reporting 
procedures, data collection procedures, and quality assurance 
procedures. Review the design produced and participate in 
design walk-throughs. 


Ensure that all facets of the project are completely visible and 
that there is close cooperation between the development team and 
the other groups with which they must interact. 


Prepare the build plan. Use unit dependency information 
provided by the technical leads and application specialists of the 
development team to specify the portions of the system that will 
be developed in each stage of the implementation phase. 
Document the capabilities to be included in each build and 
prepare a detailed milestone schedule, factoring in external 
constraints and user needs. 


Coordinate the transition to the implementation phase. It is 
usually necessary to increase the size of the development team to 
simultaneously implement multiple subsystems within a build. 
Inform the development team of the software engineering 
approaches to be used during implementation and provide the 
necessary training. Ensure that members of the development 
team understand the code and testing standards, and the quality 
assurance and configuration management procedures to be 
followed. 


Ensure that online project libraries are established, that strict 
change-control procedures concerning these libraries are 
followed, and that the necessary software for building and 
testing the system is made available so that developers can begin 
implementation immediately after the CDR. 


Direct the CDR. Schedule and participate in the review, and 
ensure that all pertinent groups take part. 


Activities of the Requirements Definition Team 


Resolve any outstanding requirements issues and TBDs, 
preparing specification modifications as necessary. Warn 
developers of any impending changes to requirements so that 
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they can plan design activities accordingly. Respond to 
developers’ questions. 


¢ Participate in design walk-throughs and the CDR. Review the 
detailed design document before CDR. During the review, 
provide a critique of the design. Use RIDs to document issues 
and discrepancies. 


Activities of the Acceptance Test Team 


* Begin work on the acceptance test plan (Section 8) as soon as 
requirements have stabilized. Requirements can be considered 
as stable when developers are no longer submitting new 
requirements questions and the number of TBDs has declined to 
a low level. 


¢ Prepare the portions of the analytical test plan that are 
needed for Build 1 (see Section 7). Meet with the development 
team to determine which analytical tests will be needed during 
the first build of the implementation phase, and complete those 
portions of the analytical test plan. 


METHODS AND TOOLS 


The same methods and tools used during the preliminary design 
phase continue in use during detailed design: 


* Functional decomposition and object-oriented design (OOD) 

e Reuse verification 

e Analysis methods: prototyping, performance modeling, and 
code analysis 

e Design walk-throughs 

e Design inspections 

* Prologs and PDL 

* SENs 


During the detailed design phase, the development team uses 
functional decomposition or OOD techniques to take the design 
diagrams down to the lowest level of detail (see Figures 5-3 and 
5-4). Prototyping and performance modeling efforts that were 
undertaken to address design issues are completed. The team also 
completes its examination of the code and documentation of reusable 
components to determine whether each unit can be incorporated into 
the system as planned. 
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The other methods and tools that continue in use from the 
preliminary design phase — design walk-throughs, design 
inspections, prologs and PDL, and SENs — have new aspects or 
applications that are introduced during detailed design. These are 


discussed in the paragraphs that follow. 


Design Walk-throughs 


During the detailed design phase, one or two 
design walk-throughs are held for each 
subsystem. Participants include members of 
the development team, the requirements 
definition team, representatives of systems 
that will interface with the software under 
development, and managers. At these walk- 
throughs, members of the development team 
step through the subsystem's design, 
explaining its algorithms, interfaces, and 
operational aspects. Participants examine and 
question the design at a detailed level to 
uncover such issues as mis-matched 
interfaces, contradictions between 
algorithms, or potential performance 
problems. 


The design of the user interface is specifically 


addressed in one or more additional walk- 


throughs. These sessions are attended by the 
future users of the system and concentrate on 
the users’ point of view. The users learn 
how the system will look to them under 
representative operational scenarios, give 
feedback to developers, and provide a "reality 
check" on the updated requirements and 
specifications. Problems encountered with 
the specifications are documented on 
question-and-answer forms and submitted to 
the requirements definition team for action. 


Design Inspections 


Design walk-throughs and 
3) | inspections are initiated 
during the preliminary design 


phase. Section 5 defines 
both inspections and 
walk-throughs and explains 
the differences between the 
two methods. 


TAILORING NOTE 


Separate, formalized design 
walk-throughs are not always held 
on small tool development efforts. 
On these projects, walk-throughs 
may be combined with design 
reviews. The reviews are generally 
informal, since the products being 
examined will not be put under CCB 
control, and focus on system 
operability and the user interface. 


Design inspections are the key methodology of the detailed design 
phase. Because the CDR is conducted primarily for the benefit of 
management and users, a detailed, component-by-component review 
of the design takes place only during design inspections. It is 
during these inspections that each separate design product is 
examined for correctness, completeness, and comprehensibility. 
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Design inspections are always conducted, regardless of the size or 
type of the project. 


At a minimum, the inspection team consists of the moderator, the 
design's author, and another member of the development team. 
Units that require detailed knowledge of the application (such as the 
physics of flight dynamics) are inspected by the development team's 
application specialist. 


On large projects, two or more of the author's peers will inspect the 
design and a representative from the quality assurance office may be 
included. Leaders of teams that are developing subsystems that 
interface with the components being inspected should also attend. 


Each inspection session covers a set of logically related units (5 to 
10 on average), for which design diagrams and unit-level PDL and 
prologs have been completed. Copies of the materials to be 
inspected are distributed to members of the inspection team several 
days before the session. 


Each member individually reviews the materials 
for technical content and adherence to design 
standards, and comes to the inspection session 
prepared to comment on any flaws he or she has 
uncovered. The moderator records the errors, 
resolves disagreement, and determines whether 
nen reinspection is necessary. The author answers 
ane ee Cas re questions and takes action items to resolve the 
modified unit must be reviewed with flaws that are identified. All participants are 
Ue Raae Maen p responsible both for finding errors and for 
not part of a "critical" thread, or its ensuring that designs are traceable to 


functions are "trivial" can be requirements. 
disastrous. 


Design inspection checklists, such as the one 
shown in Figure 6-3, are distributed to reviewers 
with the inspection materials to remind them of 
key review criteria. A master checklist is 
compiled by the moderator and is used to certify 
the unit when inspection is complete. 


Compiled Prologs and PDL 


During the detailed design phase of an Ada project, the development 
team generates and compiles all PDL for the system. For this work, 
and for the code and test activities of subsequent phases, developers 
in the STL use sets of tools and utilities that are part of an Ada 
software development environment. 


93 


Section 6 - Detailed Design 


UNIT DESIGN INSPECTION CHECKLIST 


Unit Name System Build/Release 
Initial Inspection Date 


Task Number. 


Inspection Moderator : 


KEY INSPECTION QUESTIONS Yes No Corrected 
1. Does the design present a technically valid way of achieving the 

unit's assigned function? [ 

[ 


1 g 
. Is all data required by the unit available and defined? ] [ ] 
. Is the dependency between each input and output argument and 
the processing apparent in the design? [] [] 
. Is it clear from the design where the outputs from the unit are 
generated? ] [] 
i . ls the dependency between each external reference (e.g., unit 
i file, record) and the processing apparent in the design? ] [] 
; . Does the design include the necessary error detection and 
recovery logic? [ ] 
. Is the design, as written, sufficient to handle the upper and 
lower bounds associated with unit inputs (especially arguments)? {] 


ADDITIONAL INSPECTION QUESTIONS 
8. Are the prolog and PDL consistent with the unit's design diagram? 
9. Does the PDL define logic as opposed to code? 
10. Does the prolog contain enough information to describe the unit 
clearly to the unfamiliar reader? 
11. Do both the prolog and PDL conform to applicable standards? 


Mda PL 


= ACTION ITEMS AND COMMENTS 
{List on a separate sheet. Refer to questions above by number.) 


INSPECTION RESULTS 

1. If all answers to 1-11 were "Yes, * the unit's design passes. Check 
here and sign below. 

2. If there are serious deficiencies in the design (e.g., if more than one 
key question was answered “No*) the author must correct the unit 
design and the moderator must schedule a reinspection. 

Scheduled date for reinspection: 
. If there are minor deficiencies in the design, the author must correct 
= the unit design and hold a followup meeting with the moderator. 


al LU 


[] 
[] 


[] 
[] 
[] 
[] 
[] 


mid me aal amm | g 1A C E 


Scheduled date for followup meeting: 


Moderator's signature certifies that this unit meets all applicable standards and satisfies 
its requirements, and that any identified deficiencies have been resolved (applicable at 
initial inspection, followup meeting, or reinspection). 


Moderator signature: i Date 


1 
i 


Figure 6-3. Checklist for a Unit Design Inspection 


E] 
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=e 


94 


Section 6 - Detailed Design 


DEFINITION 


In Ada, the term program library refers 
not to a library of programs, but to a 
library of compilation units that comprise 
one or more programs. To detect 
consistency errors before an entire 
program is constructed, the Ada compiler 
cross-checks information from separately 
compiled units. The program library is 
the mechanism by which the compiler 
retains the information needed to perform 
these checks efficiently. 


The Ada development environment provides a 
program library manager that functions as the 
user interface to the Ada compiler and the linker. 
The program library manager keeps track of 
which compilation units are in the library, when 
they were compiled, which ones have been 
made obsolete by more recent compilations, and 
the dependencies among units. 


The development environment also includes a 
language-sensitive editor (LSE). The LSE 
provides a template for the Ada language that 
allows developers to enter and edit prologs and 
PDL interactively. 


Ada developers use an additional tool to expedite the generation of 


package bodies. 


This utility reads a "bare-bones" package 


specification, enhances it to conform to local Ada styles (see 
Reference 22), and uses the specification to build templates for the 
package body and subprograms. 


The other components of the Ada development environment are used 
primarily during the implementation phase of the life cycle and are 
described in Section 7. 


DEFINITION 


Throughout this document, the 
term “module” is used to 
denote a collection of logically 
related units. In the flight 
dynamics environment, a 
module usually consists of 5 to 
10 units. 


TAILORING NOTE 


On Ada projects, one SEN is 
used to store documentation 
for each package. That is, the 
current listings, inspection 
checklists, and relevant design 
diagrams for the package's 
specification, body, and 
subprograms are maintained 
together in a single notebook. 


Software Engineering Notebooks {SENs) 


By the end of the detailed design phase, the 
development team's librarian will have created 
a SEN for each unit and/or module in the 
system. The developer uses the SEN to store 
all documentation that pertains to a unit, 
including the current listing of the unit's prolog 
and PDL, the unit design inspection checklist, 
design diagrams, and notes documenting 
design decisions. Through unit code, test, and 
integration, each developer retains the SENs 
for the units for which he or she is responsible. 
When the units in a module have been coded, 
tested, and certified, they are ready to be placed 
under configuration management; at this point, 
the developer gives the SENs to the librarian 
who files them in the project library. 
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MEASURES 


Objective Measures 


The objective measures used during the preliminary design phase are 
also the yardsticks used for detailed design, with one addition — 
CPU hours. The measures to be collected are as follows: 


e The number of unit designs that have been certified versus the 
number identified 

e Requirements questions and answers, TBDs, and changes 

e Staff hours 

+ Estimates of system size, effort, schedule, and reuse 

¢ Total CPU hours used to date 


The source of these data and the frequency of data collection and 
evaluation are shown in Table 6-1. The paragraphs that follow 
provide specific recommendations for evaluating these measures 
during detailed design, and supplement the guidance given in 
Section 5. 


Evaluation Criteria 


The number of TBD requirements is a vital metric in this phase. 
Ideally, all TBD requirements are resolved by the end of the phase. 
If this goal is impossible to achieve, the management team must 
assess how the remaining TBDs will affect system size, system 
design, staff hours, costs, and schedule, and then evaluate the 
feasibility of continuing. In the flight dynamics environment, 
implementation should be postponed if "mission critical” 
requirements remain unresolved or if more than 10 percent of the 
total number of requirements are still TBD. 


A large number of specification modifications in the detailed design 
phase is usually an indication that requirements and specifications 
are unstable or erroneous. The management team must assume that 
the level of specification modifications will remain high throughout 
the implementation and system test phases, and should reevaluate 
system size estimates and schedules accordingly (Figure 6-4). 


By the end of preliminary design, managers know the projected 
number of units in the system. By the end of the detailed design 
phase, all units to be reused will have been identified, along with the 
degree of modification needed to incorporate them into the new 
system. Managers can combine these new figures with productivity 
rates from past projects to reestimate the effort hours and staffing 
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Table 6-1. Objective Measures Collected During the Detailed Design Phase 


(COMLECTIANALY25) "CONTINUED | BEGUN | 
Staff hours (total and 


Weekly/monthly 
by activity) 
Weekly/biweekly 
Biweekly/biweekly 
Biweekly/biweekly 
Biweekly/biweekly 
Monthly/monthly 
Biweekly/biweekly 


levels necessary to complete development. 
(See Table 3-2 of Reference 12 for guidance 
in computing these estimates.) 


Developers and 
managers 
(via PRFs) 


Computer use (CPU Automated tool 


hours and runs) 


Requirements (changes 
and additions to 
baseline) 


Managers 
(via DSFs) 


Requirements (TBD 
specifications) 


Managers 
(via DSFs) 


Requirements 
(questions/answers) 


Managers (via DSFs) 


Estimates of total SLOC 
(new, modified, reused), 
total units, total effort , 

and schedule 


Managers (via PEFs) 


Status (units planned/ 
designed/certified) 


Managers (via DSFs) 


Recent SEL studies have shown Near the end of the phase, size estimates can 
a e Mr balloon unexpectedly if many units are moved 
of the cost to develop the unit from the "reused" to the "new" category. 
PORTHAN newly, te a percent ee a Managers need to ensure that decisions to 
projects an percent for Ada . sos 
projects. The higher percentage for Ada is create new units rather than reuse existing 
correlated eee significantly greater ones are justified and not merely a 
proportion of reused to new units on Pacea] " : 
these projects as compared with manifestation of the NIH ("not invented 
FORTRAN efforts. here") syndrome. 


Computer use, as expressed in CPU hours or the number of 
sessions/job runs, is a key indicator of progress during design and 
implementation. On a typical flight dynamics project, a small 


computer amount of CPU time should be recorded during the design phases as 

usg the development team conducts prototyping efforts and enters PDL. 
Because Ada PDL is compiled, more CPU time should be logged on 
Ada projects. 
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. TESTING TESTING | increase then drop during 


| detailed design. 


Cause: Excessive 
requirements changes and 

Management estimate ineffective change control 

of totai system size mechanisms. Requirements 
that were deleted from 

T specifications had to be 
deviorgrowthiin restored during the 
system size estimates implementation phase. 


| Corrective Actions: Assume 
| | that the instability of 
| | requirements will lead to a 

| 

| 

| 
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high level of specification 
modifications during 
implementation and testing. 
Analyze risks, replan size 
estimates accordingly, and 
request additional budget. 


Warning sign — size 
estimates wandering due to 
unstable requirements 


NOTE Inthe SEL environment, a large number of TBDs in the requirements and specifications, 
combined with a substantial number of requirements changes, typically cause a system to grow 
up to 40 percent larger than is estimated at the time of PDR. As the details of the unknown 
portions of the system become clear, the size estimate grows more rapidly. The range of 
accepted growth (shown in grey) narrows as the system becomes more defined. 


- Figure 64. Example of the Impact of Requirements Changes on Size Estimates — 
the UARS Attitude Ground Support System 


A lack of CPU hours on a project that is three-quarters of the way 
through the detailed design phase should raise a red flag. The 
management team should investigate to determine whether the team 
is avoiding using the computer because of inadequate training or is 
mired in redesign as a result of specification modifications. 


PRODUCTS 


The development team's primary product for this phase is the 
completed design for the system, including unit prologs and PDL, 
as recorded in the detailed design document. In addition, the 
management team produces a build plan for implementing the 
design, and the development team begins to develop build test plans. 
These products are described in the paragraphs that follow. 
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Detailed Design Document 


During the detailed design phase, the preliminary design report is 
expanded and refined to reflect the results of detailed design 
activities. Before CDR, this detailed design document is completed 
and distributed for review. The format and contents of the 
document are shown in Figure 6-5. 


The Build Plan 


The build plan describes the strategy that will be 
applied in constructing the system during the 
implementation phase. The plan defines the 
See SECTION 2 for definitions sequence in which software components are 
of the terms build and coded and integrated into executable subsystems 
reloasa and for guidance in and the order in which these subsystems are 


determining the number of 3 : 
builds and releases that are combined into systems. 


needed as a function of 


` t H A x 
project sze The plan usually contains three parts: 


* An itemization of the capabilities that will be 
provided in each build or release 

* The rationale, including relevant constraints, 
for providing specified capabilities in a 

Additional builds are required particular build 


on projects using Cl : : 
methodology On Clesnroor * The implementation schedule 


projects, a build should 


consist of portions of the 
software that can be readily $ meas ` ` 
tested and integrated during the preliminary design phase. During 


together and should last no £ > ; R i 
mora than? or imonthi: detailed design, the build strategy is expanded 


A preliminary build plan is usually generated 


the updated strategy is included in the SDMP and 
presented for evaluation at the CDR. 


Builds must be planned to accommodate user needs for operational 
capabilities or intermediate products. Plans must also allow for 
fluctuating and TBD requirements. Initially, the development team 
determines the optimum plan for implementing the system from a 
technical viewpoint. The management team then analyzes the plan 
and decides how it must be perturbed to accommodate the user, 
external (e.g., mission) schedules, specification modifications, and 
unknowns. Both the optimum and adjusted plans are presented at 
CDR. 
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DETAILED DESIGN DOCUMENT 


This document is the primary product of the detailed design phase. To complete the document, 
the development team updates similar material from the preliminary design report and adds 
greater detail. The suggested contents are as follows: 


1. Introduction — purpose and background of the project, overall system concepts, and 
document overview 


2. Design overview 

. Design drivers and their order of importance 

. Reuse strategy 
Discussion and high-level diagrams of the selected system design, showing hardware 
interfaces, external data interfaces, interconnections among subsystems, and data flow 

. Traceability matrix of major components against requirements and functional 
specifications 

. Design status 
(1) List of constraints, concerns, and problem areas and their effects on the design 
{2) List of assumptions and possible effects on design if they are wrong 
(3) List of TBD requirements and an assessment of their effect on system size, required 

effort, cost, and schedule 
{4) ICD status 
(5) Results of prototyping efforts 
f. Development environment 


Operations overview 
a. Operations scenarios/scripts 
b. System performance considerations 


Design description for each subsystem or major functional breakdown: 

. Overall subsystem capability 

. Assumptions about and restrictions to processing in each mode 
Discussion and high-level diagrams of subsystem, including interfaces, data flow, and 
communications for each processing mode 

. High-level description of input and output 

. Detailed description of processing keyed to operator-specified input and actions in 
terms of points of control, functions performed, and results obtained (both normal and 
abnormal, i.e., error processing and recovery) 
Structure charts or object-oriented diagrams expanded to the unit level, 
showing interfaces, data flow, interactive control, interactive input and output, and 
hardcopy output 

. Internal storage requirements, i.e., description of arrays, their size, their data capacity in 
all processing modes, and implied limitations of processing 

. Detailed input and output specifications 
(1) Processing control parameters, e.g., NAMELISTs 
(2) Facsimiles of graphic displays for interactive graphic systems 
(3) Facsimiles of hardcopy output 
List of numbered error messages with description of system's and user's actions 
Description of COMMON areas or other global data structures 
Prologs and PDL for each unit (normally kept in a separate volume because of size) 


5. Data interfaces—updated from description in preliminary design report (see Figure 5-5) 


Figure 6-5. Detailed Design Document Contents 
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Each build should address a coherent subset of the requirements and 
specifications and take three to five months to implement. Each 
build should cover a set of completed units; that is, the build plan 
should not require modifications or enhancements to individual units 


during later builds. 


TAILORING NOTE 
(cont.) 


TAILORING NOTE 


in the flight dynamics 
environment, the initial build 
{B1) usually provides the core 
capabilities needed in a function- 
ing system. Middle builds (B2 to 


The first build must be kept simple, 
particularly if the development team 
is unfamiliar with the development 
environment or programming 
language being used. The next 
builds should address high-risk 
specifications and critical software 
capabilities, such as performance 
requirements, major control 
functions, and system and user 


Bn-1) supply all critical 
capabilities. The last build (Bn) 
is restricted to "bells and 

whistles” and problem fixes. 


interfaces. The build strategy must 
ensure that capabilities that can 
have a major impact on the 
software design are completed 
early, so that any problems can be 
handled while there is still time to 
recover. 


Since the last build will grow to include the 
implementation of specification modifications 
and the resolution of problems remaining from 
earlier builds, it should be kept small in initial 
for detailed information about the planning. The next-to-last build, therefore, must 
purpose and content of these plans. supply all crucial system capabilities. If 
IE specification modifications that add new features 

to the system are received during the 
implementation phase, additional builds that 
extend the phase may be needed to ensure that 
existing builds can proceed on schedule. 


The build plans for flight dynamics 
projects also include the dates by 
which developers need to receive 
unit-level, analytic test plans from the 
acceptance test team. See SECTION 7 


Build Test Plans 


As soon as a build has been defined, the development team can 
begin to specify the tests that will be conducted to verify that the 
software works as designed and provides the capabilities allocated to 
the build or release. The development team executes the build test 
plan immediately following integration of the build. 


The test plan for the first build is defined during the detailed design 


phase. Any modifications to the overall testing strategy that are made 
as a result of defining this first test plan are presented at the CDR. 
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Test plans for the remaining builds are generated during the 
implementation phase. 


The format and content of build test plans are described in Section 7. 


CRITICAL DESIGN REVIEW 


The detailed design phase culminates in the CDR. This review is 
attended by the development team and its managers, the 
requirements definition team and its managers, quality assurance 
representatives, user representatives, the CCB, and others involved 
with the system. Participants evaluate the detailed design of the 
system to determine whether the design is sufficiently correct and 
complete for implementation to begin. They also review the build 
plan to ensure that the implementation schedule and the capabilities 


allocated to the builds are feasible. 


The emphasis at CDR is on modifications 
— to requirements, high-level designs, 
system operations, and development plans 
— made since the PDR. Speakers should 
highlight these changes both on the slides 
and during their presentations, so that they 
become the focus of the review. The CDR 
also provides an opportunity for the 
development team to air issues that are of 
concern to management, the mission project 
office, quality assurance personnel, and the 
CCB 


Figure 6-6 shows the recommended CDR 
format. An outline and suggested contents 
of the CDR hardcopy material are presented 
in Figure 6-7. Note that material that was 
covered at PDR is not presented again, 
except as needed to contrast changes. For 
this concise format to be effective, 
participants must be familiar with the project 
background, requirements, and design. 
They should have attended the PDR and 
studied the detailed design document before 
the meeting. 


Reviewers should address the following questions: 


TAILORING NOTE 


For very large projects, a CDR 
should be held for each major 
subsystem and/or release in 
order to cover all aspects of the 
system and to accommodate 
changing requirements. On such 
projects, it is vital to have one 
review, Le., a System Design 
Review, that covers the entire 
system at a high level. 


REUSE NOTE 


PLA 


* Does the design satisfy all requirements und specifications? 
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At the CDR, developers present 
statistics showing the number 
and percentage of components to 
be reused, and which of these are 
drawn from the RSL. They also 


present key points of the detailed 
reuse strategy, identify any 
changes to the reuse proposal 
that have been made since PDR, 
and describe new/revised reuse 
tradeoff analyses. 
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CDR FORMAT 


Presenters — software development team 


Participants 

» Requirements definition team 

e Quality assurance representatives from both teams 
» Customer interfaces for both teams 

e User representatives 

» Representatives of interfacing systems 

e System capacity/performance analysts 

»CCB 


Attendees should be familiar with the project background, require- 
ments, and design. 


Schedule — after the detailed design is completed and before 
implementation is begun 


Agenda — selective presentation of the detailed design of the system. 
Emphasis should be given to changes to the high-level design, system 
operations, development plan, etc. since PDR. 


Materials Distribution 

e The detailed design report is distributed at least 10 days before the 
CDR. 

e Hardcopy material is distributed a minimum of 3 days before the 
review. 


Figure 6-6. CDR Format 


¢ Are the operational scenarios acceptable? 


+ Is the design correct? Will the transformations specified produce 
the correct output from the input? 


+ Is the design robust? Is user input examined for potential errors 
before processing continues? 


e Have all design guidelines and standards been followed? How 
well have data usage and access been localized? Has coupling 
between units (i.e., interunit dependency) been minimized? Is 
each unit internally cohesive (i.e., does it serve a single 
purpose)? 


+ Ts the design testable? 
e Is the build schedule structured to provide early testing of end- 


to-end system capabilities? Is the schedule reasonable and 
feasible for implementing the design? 
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HARDCOPY MATERIAL FOR THE CDR 


Agenda — outline of review material 


introduction — background of the project, purpose of the system, and an agenda 
outlining review materials to be presented 


Design overview — major design changes since PDR {with justifications) 

a. Design diagrams, showing products generated, interconnections among subsystems, 
external interfaces 

b. Mapping of external interfaces to ICDs and ICD status 


Results of prototyping efforts 


Changes to system operation since PDR 
a. Updated operations scenarios/scripts 
b. System performance considerations 


Changes to major software components since PDR (with justifications} 


7. Requirements traceability matrix mapping requirements to major components 


& Software reuse strategy 
a. Changes to the reuse proposal since PDR 
b. New/revised reuse tradeoff analyses 
c. Key points of the detailed reuse strategy, including software components to be reused 
in future projects 
d. Summary of RSL contributions — what is used, what is not, reasons, statistics 


9. Changes to testing strategy 
a. How test data are to be obtained 
b. Drivers/simulators to be built 
c. Special considerations for Ada testing 


10. Required resources — hardware required, internal storage requirements, disk space, 
impact on current computer usage, impacts of compiler 


Changes to the SDMP since PDR 


Implementation dependencies (Ada projects) — the order in which components 
should be implemented to optimize unit/package testing 


Updated software size estimates 


Milestones and schedules including a well-thought-out build plan 


15. Issues, risks, problems, TBD items 
a. Review of TBDs from PDR 
b. Dates by which TBDs and other issues must be resolved 


Figure 6-7. CDR Hardcopy Material 
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EXIT CRITERIA 


To determine whether the development team is ready to proceed with 
implementation, the management team should consider the following 
questions: 


FINISH 


Are all design diagrams complete to the unit level? Have all 
interfaces — external and internal — been completely specified? 


* Do PDL and prologs exist for all units? Have all unit designs 
been inspected and certified? 


e Have all TBD requirements been resolved? If not, how will the 
remaining TBDs impact the current system design? Are there 
critical requirements that must be determined before 
implementation can proceed? 


e Have the key exit criteria for the phase been met? That is, has 
the detailed design document been completed, has the CDR been 
successfully concluded, and have responses been provided to all 
CDR RIDs? 


When all design products have been generated and no critical 
requirements remain as TBDs, the implementation phase can begin. 
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UIREMENTS: | PRELIMI-| DETAILED 
CYCLE REQUIRE ENTS R ENTS: | NARY DESIGN 
PHASES cee ANALYSIS | DESIGN 


SECTION 7 


THE IMPLEMENTATION PHASE 


ENTRY CRITERIA 


* Detailed design document generated 
« CDR completed 
e COR RIDs answered 


+ System code and supporting data 

¢ Build test plans and results 

* System and analytical test plans 
SSSI TE ERIE SS EIT ee 


MEASURES 


* Units coded/ code-certified/ test-certified 
vs. units identified - 

* Requirements Q&As, TBDs, and changes 

» Estimates of system size, effort, 
schedule, and reuse 

¢ Staff hours 

« CPU hours 

* SLOC in controlled libraries (cumulative) 

* Changes and errors (by category) 


« Code reading 

. Unit testing 

* Module integration testing 
* Build testing 

e Configuration management 
* SENs 

* CASE 


PRECEDING PASE BLANK NOT FILMED 


RAE 
LI 


EXIT CRITERIA 


«All system code and supporting data 
generated and tested 

+ Build test plans successfully executed 

* System test plan completed 

« User's guide drafted 


KEYA 


Requirements Definition Team 
«Resolve any remaining requirements issues 
* Participate in build design reviews (BDRs) 


Development Team 
* Code new units and revise existing units 
« Read new and revised units 
* Test and integrate each unit/madule* 
* Plan and conduct build tests* 
+ Prepare the system test plan* 
« Draft the user's guide 
* Conduct build design reviews 


Management Team 


+ Reassess schedules, staffing, training, and 
other resources 

e Organize and coordinate subgroups 
within the development team 

e Control requirements changes 

e Ensure quality in processes and products 

+ Direct build design reviews 

* Coordinate the transition to system testing 


H Acceptance Test Team 


» Complete the acceptance test plan draft 
* Complete the analytical test plan 


LR AEH ERIE INRIA SIAR BEER OESE REEE OOO KANAS AOE EEEE OOA ODN 
*Projects that use the Cleanroom methodology have an independent test team. 
Developers prepare and verify each other's code, then submit the code (in small builds) 
to the test team for compilation and testing. 
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OVERVIEW 


The purpose of the implementation phase is to build a -omnlete, 
high-qualiiy software system from the "blueprint" provided in the 
detailed design document. The implementation phase begins after 
CDR and proceeds according to the build plan prepared during the 
detailed design phase. For each build, individual programmers code 
and test the units identified as belonging to the build, integrate the 
units into modules, and test module interfaces. 


At the same time, the application specialists on the development team 
prepare plans designed to test the functional capabilities of the build. 
Build regression tests — a selection of tests already conducted in 
previous builds — are included in each build test plan to ensure that 
newly added capabilities have not affected functions implemented 
previously. All build test plans are reviewed for correctness and 
completeness by the management team. 


When all coding, unit testing, and unit integration testing for the 
build are complete, selected members of the development team build 
the system from the source code and execute the tests specified in 
the build test plan. Both the management team and the development 
team review the test results to ensure that all discrepancies are 
identified and corrected. l 


As build testing progresses, the development team begins to put 
together the user's guide and the system description documents. A 
draft of the user's guide must be completed by the end of the 
implementation phase so that it can be evaluated during system 
testing. Material from the detailed design document is updated for 
inclusion in the system description document, which is completed at 
the end of the system test phase. 


Before beginning the next build, the development team conducts a 
build design review (BDR). The formality of the BDR depends on 
the size of the system. Its purpose is to ensure that developers, 
managers, and customer representatives are aware of any 
specification modifications and design changes that may have been 
made since the previous review (CDR or BDR). Current plans for 
the remaining builds are presented, and any risks associated with 
these builds are discussed. 


The plans for testing the completed system (or release) are also 
generated during the implementation phase. Application specialists 
from the development team prepare the system test plan, which is 
the basis for end-to-end testing during the next life cycle phase. At 
the same time, members of the independent acceptance test team 
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prepare the test plan that they will use during the acceptance test 
phase. 


The implementation process for a single build is shown in 
Figure 7-1. Figure 7-2 shows that these implementation processes 
are repeated for each build and that a larger segment of the life cycle 
— extending from the detailed design phase through acceptance 
testing — is repeated for each release. 


PROLOGS 
AND POL 


CODED 
UNITS* 


CONDUCT 

BUILD DESIGN 
REVIEW 

71 


APPROVED 
DESIGN 
CHANGES 


EUSTING CONFIGURED SOFTWARE LIBRARIES 
UNITS 


CODE OR MODIFY 
UNITS: 


REUSABLE SOFTWARE LIBRARY 


UNIT SOURCE 
CODE LISTINGS 


DETALED DESIGN 
DOCUMENT 


LINKS TO ASL 
(ior verbatim reuse) 


VERIFY COOgO 
UNITS* 
73 


UPDATED REQUIREMENTS 
AND SPECIFICATIONS 


TESTED ANO 
INTEGRATED 
UNITS 


TESTED ANO 
INTEGRATED 
UNITS 


BUILD PLAN PREPARE BUILD = 
TE Srang a BUILD TEST PLAN 
Se 74 


*NOTE: Verification usually consists of code reading and unit testing. After the programmer 
compiles a unit successfully, it is read by at least two other members of the deveiopment team. 
When the readers pass and certify the unit, the programmer conducts the unit tests. In the BUILD TEST REPORT 
Cleanroom methodology, however, coded units are read, then submitted to an independent test 
team for compilation, integration, and testing. 


See KEY ACTIVITIES for descriptions of tha processes in this diagram. The contents of build test 
plans are covered under PRODUCTS. BUILD REVIEWS are the topic of a separate subsection. 


Figure 7-1. Implementing a Software Build 
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*NOTE: See Builds and Releases in Section 2 for guidance on the number af 
builds/relaases appropriate to projects of varying size and complexity 


** NOTE: A build design review (BOR) is haid for every build except the first. 
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Figure 7-2. Phases of the Life Cycle Are Repeated for Multiple Builds and Releases 


KEY ACTIVITIES 


Although the activities of coding, code reading, unit testing, and 
integrating a single module are conducted sequentially, the 
development team implements each module in paralle! with others. 
Unless the project is very small, the development team is partitioned 
into multiple groups during the implementation phase. Each group 
is assigned to develop one or more subsystems, and group members 
are each assigned one or more modules. During a given build, the 
group will code, read, and tes! the modules of its subsystem that are 
scheduled for the build. Therefore, coding, code reading, unit 
testing, and module testing activities may be conducted 
simultaneously at any given time. 


During the implementation phase, the application specialists in the 
development team do not code and test units. Instead, they apply 
their expertise by reading the code of selected units (such as those 
with complex, application-specific algorithms) and inspecting unit 
and module test results. They also prepare the system test plan. 


The key technical and managerial activities of the implementation 
phase are summarized below and shown on a timeline in Figure 7-3. 
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Activities of the Development Team: 


* Code new units from the detailed design specifications and 
revise existing units that require modification. Code units so 
that each PDL statement can be easily matched with a set of 
coding statements. Use structured coding principles and local 
coding conventions (References 22 and 23). Prepare the 
command language (e.g., JCL or DCL) procedures needed to 


execute the units. 


DEFINITION 


Code reading is a systematic procedure 
for inspecting and understanding source 
code in order to detect errors or 
recommend improvements. It is 
described in detail in this section under 
METHODS AND TOOLS. Certification is 
a part of the quality assurance process 
wherein an individual signs a checklist 
or form as an independent verification 
that an activity has been successfully 
completed. 


See METHODS AND TOOLS 
in this section for more 
detailed information on 
coding standards and unit 
testing. Test plans — 
analytic test plans, build test 
plans, and the system test 
plan — are described under 
PRODUCTS. 


On most projects, unit and module tests 
need not be performed separately. As 
more units for a particular module are 
developed, they are unit tested in the 
context of the units previously developed 
within the module. 


Units may be either executable or data. 
On Ada projects, the module takes the 
form of a package. 


Read new and revised units. Ensure that 
each unit is read by a minimum of two 
members of the development team who are 
not the unit's authors. Correct any errors that 
are found, reinspecting the unit as necessary. 
Certify all unit code. 


Test each unit and module. Prepare unit test 
procedures and data, and conduct unit tests. If 
the acceptance test team has provided an 
analytical test plan for the unit, complete the 
test cases specified in the plan and verify that 
the computational results are as expected. 
Have an experienced member of the 
development team review and certify all unit 
test procedures and results. 


Integrate logically related units into modules 
and integrate the modules into the growing 
build. Define and run enough tests to verify 
the I/O generated by the module and the 
interfaces among units within the module. 
Ensure that the results of module testing are 
reviewed and certified. 


Plan and conduct build tests. Prepare the test 
plan for the build, and complete the 
preparation of command procedures and data 
needed for build testing. 
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Resolve any remaining requirements issues and TBDs 


~ Participate in BDRs 


Code new units and revise reused units 
Read and certify new and revised units 
Test and integrate each unit and module 


Plan and conduct build tests 


Conduct BDRs 


Prepare the system test plan 


Draft the user's guide 


ee a 


Prepare the analytical test plan 


Refine the acceptance test plan 


Vv 


ee 
Record project history data; reassess schedules, staffing, resources 


Organize and coordinate implementation groups 


Control requirements changes; ensure quality 


y 


Direct BDRs 


Update SDMP estimates 


Coordinate the transition to system testing 
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Figure 7-3. Timeline of Key Activities in the implementation Phase 
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The load module (or executable image) for the build is created by 
the project librarian. When the load module is prepared, execute 
the tests specified by the test plan for the build. Ensure that all 
output needed for test evaluation is generated. Record any 
discrepancies between the results specified in the plan and actual 
results. 


Correct all discrepancies that are found. When the affected units 
are repaired and tested, file a report of the changes with the 
project librarian. The librarian ensures that the configured 
libraries and test executables are updated with the revised units. 


Rerun any tests that failed, and verify that all errors have been 
corrected. When all build tests have been successfully 
completed, prepare a written report of the test results. 


* Prepare the system test plan for use during the system testing 
phase. Begin to develop the plan immediately after CDR, so that 
it will be ready by the end of the phase. Prepare the command 
procedures and input data needed for system testing. 


* Prepare a draft of the user's 
guide, using sections of the 
detailed design document (the 
operations overview and design 
description) as a foundation. 


Begin work on the system 
description document by updating 
data flow/object diagrams and 
structure charts from the detailed 


See BUILD design document. 
The format and contents of the DESIGN REVIEWS 8 
user's guide and system 


in this section for 
description documents are _ aoe * Conduct a BDR before every 
karnicad undar PRODUCTS iR VeA TOR Rd build except the first (changes to 
content of BDRs. the design and/or build plan that 
apply to the first build are covered 
during the CDR). Ensure that all 
design changes are communicated 
to development team members, 
users, and other participants. 
Present the key points of the build 
plan, making certain that all 
participants understand their roles 
in the build, the schedule, and the 
interfaces with other groups or 
activities. 


113 


Section 7 - Implementation 


Activities of the Management Team 


e Reassess schedules, staffing, training, 
and other resources. At the beginning of 


the phase, record measures and lessons REUSE NOTE 
learned from the detailed design phase and $ AN os 
. . . anagers snoula make 
add this information to the draft of the frequent checks throughout the 
: software development history. As design and implementation 
: : : : phases to ensure that reuse is 
: implementation progresses, use the size of hot being compromised for 
completed units to refine estimates of total short-term gains in schedule or 
; system size. Use measures of actual budget. Managers must 
| ; actively promote the reuse of 
i resources expended and progress during existing software and stress 
the implementation phase to update cost the importance of developing 
software that is reusable in the 
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and resource estimates. (See Measures future: 
subsection and Reference 12.) aa 


Reestimate system size, effort required to complete, schedules, 
and staffing each time a build is completed. Toward the end of 
the implementation phase, update the SDMP with effort, 
schedule, and size estimates for the remaining phases. 


Organize and coordinate subgroups within the development 
team. At the beginning of the phase, organize the development 
team into small, three- to five-person groups. Assign each 
group a cohesive set of modules to implement. Regardless of 
the size of a module set, the same group should code and test the 
units and the integrated modules. As much as possible, ensure 
that the designer of a unit is also responsible for its coding and 
verification. 


Ensure that any new personnel joining the project during this 
phase are adequately trained in the standards and procedures 
being followed (including data collection procedures) and in the 
development language and toolset. Make experienced personnel 
available to direct new and/or junior personnel and to provide 
on-the-job training. 


Control requirements changes. Thoroughly evaluate the 
impacts of any specification modifications received during this 
phase. Report the results of this analysis to the requirements 
definition team and the customer. 


Ensure that customers and users of the system agree on the 
implementation schedule for any specification modifications that 
are approved. To minimize their impact on the build in 
progress, schedule the implementation of new features for later 
builds. 
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Ensure quality in processes and products. Make spot checks 
throughout the phase to ensure adherence to configuration 
management procedures, quality assurance procedures, coding 
standards, data collection procedures, and reporting practices. 
Configuration management procedures — especially change 
control on the project's permanent source code libraries — are 
critical during the implementation phase when the staff is at its 
peak size and a large amount of code is being produced. 


Monitor adherence to the build plan. Know at all times the 
status of development activities and the detailed plans for 
development completion. 


Review the draft user's guide and system test plans. Participate 
in the inspection of test results for each build and assist the 
development team in resolving any discrepancies that were 
identified. 


Direct all BDRs, and ensure that any issues raised during the 
reviews are resolved. 


Coordinate the transition to the system testing phase. Staff 
the system test team with application specialists, and include one 
or two analysts to take responsibility for ensuring the 
mathematical and physical validity of the test results. (This will 
also guarantee that some analysts are trained to operate the 
software before acceptance testing.) Assign a lead tester to 
direct the system testing effort and to act as a final authority in 
determining the success or failure of tests. 


Ensure that the data and computer resources are available to 
perform the steps specified in the system test plan. Inform 
personnel of the configuration management and testing 
procedures to be followed and provide them with the necessary 
training. 


At the conclusion of the phase, hold an informal system test 
readiness review (STRR). Use this meeting to assess whether 
the software, the system test team, and the test environment are 
ready to begin testing. Assign action items to resolve any 
outstanding problems and revise schedules accordingly. 


Activities of the Requirements Definition Team 


Resolve any remaining requirements issues. If implemen- 
tation is to proceed on schedule, all TBD requirements must be 
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resolved early in the phase. If any requirements cannot be 
defined because external, project-level information (¢.g., 
spacecraft hardware specifications) is incomplete, notify upper 
management of the risks and the potential impact to development 
schedules. Obtain deadlines by which the missing information 
will be supplied, and work with the development team to adjust 
schedules accordingly. Prepare a plan to mitigate these risks and 
reduce the possible schedule delays and cost overruns. 


Ensure that changes to requirements that are of external origin 
(e.g., changes to spacecraft hardware) are incorporated into 
specification modifications without delay. Submit all 
specification modifications to the management team for technical 
evaluation and costing. 


* Participate in all BDRs. Warn the development team of 
potential changes to requirements that could impact the design or 
otherwise affect current or remaining builds. 


Activities of the Acceptance Test Team 


* Complete the draft of the acceptance test 
plan that was begun during the detailed 
design phase. The draft should be 
provided to the development team before 

A recommended outline of 


the start of system testing. IÆ the acceptance test plan is 
provided under PRODUCTS 

* Prepare the analytical test plan. At the in Section 8. 
beginning of a build, supply the 
development team with the parts of the 
analytical test plan that they will need 
during the build to verify the results of 
complex mathematical or astronomical 


computations. 


METHODS AND TOOLS 
The key methods and tools of the implementation phase are 


e Code reading 

¢ Unit testing 

¢ Module integration testing 
¢ Build testing 

e Configuration management 


+ SENs 
. CASE 
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The SEL requires the use of 
structured coding principles 
and language-dependent 
coding standards. SEL 


coding standards are 
documented in References 
22 (Ada) and 23 (FORTRAN). 
Reference 24 is one of the 
many sources of information 
on structured programming. 


The SEL's recommendation that 
at least two code readers 
examine each unit stems from a 
Cleanroom experiment 
(Reference 3). This project 
discovered that an average of 
only 1/4 of the errors in a unit 
were found by both readers. 
That is, 75% of the total errors 
found during code reading were 
found by only one of the readers. 


Some compilers allow the user to 
generate a cross-refarence listing 
showing which variables are used 
in the unit and their locations. 
Code readers should use such 
listings, if available, to verify that 
each variable is initialized before 
first use and that each is 
referenced the expected number 
of times. Unreferenced variables 


may be typos. 
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Each is discussed below. 
Code Reading 


The first step in the unit verification process is 
code reading, a systematic procedure for 
examining and understanding the operation of a 
program. The SEL has found code reading to 
be more cost effective in uncovering defects in 
software than either functional or structural 
testing and has formalized the code reading 
process as a key implementation technique 
(References 25 and 26). 


Code reading is designed to verify the logic of 
the unit, the flow of control within the unit, and 
boundary conditions. It is performed before 
unit testing, not afterwards or concurrently. 
Only code that has compiled cleanly should be 
presented for code reading. 


Every new or modified unit is read by two or 
more team members. Each reader individually 
examines and annotates the code, reading it line 
by line to uncover faults in the unit's interfaces, 
control flow, logic, conformance to PDL, and 
adherence to coding standards. A checklist that 
is used by code readers on SEL-monitored 
projects is shown in Figure 7-4; its use fosters 
consistency in code reading by ensuring that 
the reader has a list of typical errors to look for 
and specific points to verify. 


The readers and the unit's developer then meet 
as a team to review the results of the reading 
and to identify problems that must be resolved. 
They also inspect the test plan for the unit. If 
errors have been discovered in the unit, the 
reader who is leading the meeting (the 
moderator) returns the unit to the implementor 
for correction. The unit is then reexamined. 
When all errors have been resolved, the 
moderator certifies that the code is satisfactory 
and signs the checklist. The implementor files 
the certified checklist in the SEN for the unit. 
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UNIT CODE INSPECTION CHECKLIST 


System Build/Release 
Initial Inspection Date 


Unit Name 


Task Number. 


Inspection Moderator. 


KEY INSPECTION QUESTIONS Yes No Corrected 
1. Is any input argument unused? Is any output argument 


not produced? [] [] [] 
2. Is any data type incorrect or inconsistent? [] [] [] 
3. Is any coded algorithm inconsistent with an algorithm 

explicitly stipulated in PDL or in requirements/specifications? [] [] [] 
4. Is any local variable used before it is initialized? [] [] [] 
5, Is any external interface incorrectly coded? That is, 

is any call statement or file/database access 

incorrectly coded? Also, for an Ada unit, is any external 

interface not explicitly referenced/with'd-in? [] { ] L] 
6. Is any Jogic path incorrect? [] [] [] 
7. Does the unit have multiple entry points or multiple, normal 

(non-error) exits? [] [] [] 


ADDITIONAL INSPECTION QUESTIONS 
8. Is any part of the code inconsistent with the unit design 


specified in the prolog and PDL? [] [] [] 
9. Does the code or test plan contain any unauthorized 
deviations from project standards? [] [] [] 
10. Does the code contain any error messages that might be 
unclear to the user? [] [] [] 
11. If the unit was designed to be reusable, has any hindrance to 
reuse been introduced in the code? [] [] [] 


ACTION ITEMS AND COMMENTS 
_ (List on a separate sheet. Refer to questions above by number.) 


INSPECTION RESULTS 
1. If all answers to 1-11 were "No, * the unit's code passes. Check 
here and sign below. [ ] 
2. If there are serious deficiencies in the code (e.g., if more than one 
key question was answered "Yes") the author must correct the unit 
design and the moderator must schedule a reinspection. 
Scheduled date for reinspection: i 
3. If there are minor deficiencies in the code, the author must correct 
the unit design and hold a followup meeting with the moderator. 
Scheduled date for followup meeting: 2 


Moderator's signature certifies that this unit meets all applicable standards and satisfies 
its requirements, and that any identified deficiencies have been resolved (applicable at 
initial inspection, followup meeting, or reinspection}. 


Moderator Signature: Date: 


Figure 7-4. Sample Checklist for Code Inspection 
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Each member of the development team should be assigned units to 
read. If only one or two developers are appointed to read all the 
units, the other team members will lose an opportunity to gain 
expertise and increase their understanding of the system. 


The code reader for a particular unit should not be selected by the 
unit's developer, but by the task leader. The choice of code reader 
should be appropriate to the character, complexity, and criticality of 
the unit. For example, units that contain physical or astronomical 
calculations should be read by application specialists who are 
familiar with the requirements and able to uncover analytical errors. 
Likewise, control units that use operating system services should be 
read by an operating system expert, and those that interface with a 
DBMS should be examined by a database specialist. 


Unit Testing 


Unit testing is the second step in verifying the logic, functionality, 
computations, and error handling of a unit. The intent of unit testing 
is to confirm that the unit provides the capability assigned to it, 
correctly interfaces with other units and data, and is a faithful 
implementation of the unit design. 


In general, the developer who coded the unit executes the tests 
identified in the unit test plan; independent testers are not required 
unless the unit must comply with stringent safety or security 
requirements. 


The test plan should be tailored for the type of 
unit being tested. Structural (path) testing is 
critical for units that affect the flow of control 
through the system. The test plan for such a 
unit is generated by the developer from the 
unit's design and should include a sufficient 


On projects employing the 
Cleanroom methodology, no 
testing is conducted at the unit 
level. When a unit has been read 


and certified, it is submitted to an 
independent test team for 
compilation, integration, and 
functional testing. The tests that 
are conducted are a statistically 
selected subset of system tests. 


number of test cases so that each logic path in 
the PDL is executed at least once. For units 
whose function is primarily computational, the 
developer may execute an analytical test plan. 
Analytical test plans are prepared by the 
acceptance test team to assist developers in 
verifying the results of complex mathematical, 
physical, and astronomical calculations (see 
Products). Units that are part of the user 
interface are tested using yet another approach 
— one that ensures that each of the user 
options on the screen is thoroughly exercised. 
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When unit testing is complete, the test results 
are reviewed by the developer's team leader or 
application, specialist. The reviewer certifies The use of a symbolic 


the completeness and correctness of the test. debugger can greatly 
improve the efficiency of 


That is, he or she checks the results against the 
test plan to ensure that all logic paths have been 
tested and verifies that the test results are 
accurate. As with code reading, use of a 
checklist is recommended to assist reviewers 
and maintain consistency. 


The unit test plan and test results are maintained in the SEN for the 
unit. If extensive changes are made to the unit at a later time, the 
unit code must be reread, and the unit must be retested and certified. 


The management team determines the level of rigor in unit testing 
that is most cost effective for the project. For example, in some 
projects it may be more efficient to conduct testing at the module 
level than to test individual units. Indeed, for Ada projects, unit 
testing should generally be conducted within the context of the 
module (i.e., Ada package). 


Module Integration Testing 


Developers integrate individual, tested units into modules, then 
integrate these modules into the growing build. The method of 
integration testing that is used should be appropriate to the design of 
the system. Menu-driven systems, for example, lend themselves to 
either top-down or thread testing (Figure 7-5). In contrast, systems 
with complex, computational utilities may benefit from a bottom-up 
approach. As in unit testing, integration test plans and results are 
reviewed and certified by other members of the development team. 


In the SEL environment, modules are verified using the existing, 
previously tested build as a test bed. Units not yet implemented 
exist in the module as stubs; that is, they contain no executable 
instructions except to write a message that the unit was entered and 
has returned control to the calling unit. This approach tests both the 


module's integration into the growing system and the internal code _ 


of the units that it comprises. Test drivers, which must themselves 
be coded and tested, are thus eliminated, and higher-level modules 
are exercised more frequently. 


When a module has been integrated, tested, and certified, the 
developer completes a component origination form ( COF) for each 
of the units in the module. This SEL form has a dual function. The 
information provided on the form is stored in the SEL database and 
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unit testing. The output 
generated by the symbolic 
debugger is filed in the SEN 
for the unit. 
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TOP-DOWN TESTING THREAD TESTING 


PaP fp 


` 


O Previously integrated units 
KFA Units integrated this iteration 


O Software stubs 


Top-down testing integrates additional modules level by level. Thread testing 
builds a single end-to-end path that demonstrates a basic functional capability, 
then adds on to that. 


Figure 7-5. Integration Testing Techniques 


used to track system composition, growth, and change throughout 
the project's life cycle. The form is also a key configuration 
management tool for the project librarian, who uses the source file 
information on the form to enter completed units into the project's 
controlled software library. 


Build Testing 


After all modules in the build have been coded and tested, the 
development team conducts build tests on the software in the 
controlled library. The purpose of build testing is to verify that the 
software provides the functionality required of the build and is a 
correct implementation of the design. Build regression tests are also 
conducted to ensure that functions provided by previous builds have 
not been adversely affected by the new components. 


Build tests are executed by selected members of the development 
team following a formal build test plan (as described under Products 
in this section). The project librarian builds the executable images of 
the software from the tested modules in the configured library. The 
testers generally use a test checklist or report form to record the 
results of each test case as it is executed. 


The results of tests are rigorously evaluated by developers, 


application specialists, and the management team. Operational 
difficulties, abnormal terminations, and differences between actual 
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and expected test results are recorded on special report forms. discrepancy 
These discrepancy reports are used to ensure that each problem that reports 
is observed is resolved. 


The development team locates the cause of each discrepancy and 
corrects the appropriate units. For each logical change that is made 
: to controlled software, developers submit a change report form change report 
i (CRF). This SEL form is used to gather information about the form (CRF) 
character of the changes made, their source, the effort required, and 


A 


the number of changes due to errors. 


Configuration Management and SENs 


During the implementation phase, adherence to 
configuration management procedures 
becomes critical. Source code is generally 
placed under configuration control one module 
at a time. When the units in the module have 
been coded, tested, and certified, the developer 
submits COFs for the units to the project 
librarian. The librarian moves the units into 
the project's configured source code libraries 
and files the units' SENs. Any further 
changes to these units must be approved by the 
development team leader or designated 
application specialist. The developer must 
check out the appropriate SENs, update the 
unit(s), fill out one or more CRFs, and update 
and return the SENs to the project library. 
Changes to configured libraries are made 
solely by the project librarian, who replaces 
configured units with the updated versions. 


of discrepancies that must be 
rectified can be substantial. 

RM] Managors must track these 
discrepancies, assign personnel 
to resolve them, set dates for 
resolution, and verify that all 
discrepancies have been 
corrected. Use of a tracking 
tool, such as CAT (Reference 
27) or a PC-based DBMS, makes 

this task easier. 


& For large systems, the number 


mr mais 
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The project librarian maintains the central project library, adding to it 
all documentation produced during the implementation phase: SENs 
for completed units/modules, test plans and results for each build, 
= drafts of the user's guide and system test plan, and system 
description information. The librarian also catalogs and stores 
CRFs for any changes made to software under configuration 
management, and files specification modifications and updates to 
design documents. 


3 The management of a project's controlled source code libraries can 
be greatly facilitated by the use of an online configuration 
management tool. In the flight dynamics environment, DEC's Code 
Management System (CMS) is used to manage software developed 
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in the STL's VAX environment. PANVALET and CAT are used 
for systems developed, operated, and maintained in the IBM 
environment of the FDF (Reference 27). 


Configuration management tools can be used to store all code, test 
drivers, data, and executable images; to track changes from one 
version to the next; and, most importantly, to provide access 
control. CMS, for example, allows the developer or librarian to 
reconstruct any previous version of a library element, tracks who is 
currently working on the element, and maintains a record of library 
access. With a configuration management tool, the project librarian 
can readily maintain multiple versions of the system, called 
baselines, each of which represents a major stage in system 
development. Baselines are generally established for each build, for 
system testing, for acceptance testing, and for operational use. 


CASE 

Use of CASE tools can yield significant benefits during the 
implementation phase. The following tools are those that have been 
found to be most beneficial in the SEL's development environment. 


Language-sensitive editors, such as VAX LSE, provide language- 


language- 3 > 
sensitive specific templates that help the programmer to enter and compile 
editors code efficiently, to review resultant diagnostic messages, and to 


correct errors — all within a single editing session. Debuggers 
allow the developer to suspend execution of a program, locate and 
correct execution errors, and return to program execution 
interactively. 


Static code analyzers, such as the VAX Source Code Analyzer 
(SCA), the RXVP80 static analyzer, and the Static FORTRAN 
Source Code Analyzer Program (SAP), provide cross-referencing 
capabilities among source files. They allow the developer to locate 
subprograms, variables, and data references and to answer 
questions such as "in which units is variable X used?". Additional 
functions provided by some analyzers include the display of call- 
trees and the extraction of design information. 


static code 
analyzers 


Performance analyzers (e.g., the VAX Performance and Coverage 
Analyzer or Boole & Babbage's TSA/PPE) help the developer 
examine the run-time behavior of software to locate inefficiencies 
and bottlenecks. They collect data and statistics during the execution 
of a program and can generate histograms, tables, and call-trees 
from the data. Performance analyzers can also help locate portions 
of the software that have not been executed during testing. 


performance 
analyzers 
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Compilation systems (e.g., Alsys' UNIX Ada 
Compilation System or the VAX DEC/Module 
Management System) automate and simplify 
the process of building complex software 
applications. Compilation systems access 
source files in the program library and follow 
the sequence of dependencies among the files 


compilation 
systems 


Ada code development and 
compilation tools are 
described under METHODS 


to automatically build the system from current AND TOOLS in Section 6. 


versions. This allows a developer or project 


Performance analyzers and 
static code analyzers are also 


librarian to rebuild a system using only discussed in Section 5. 


components that were changed since the 
previous system build. 


In the FDF, a tailored software development environment called 
SDE gives developers access to a variety of tools and utilities. SDE 
(Reference 27) integrates editors, compilers, and file allocators 
under a single, menu-driven framework. It is a customization of 
IBM's Interactive System Productivity Facility (ISPF) that provides 
additional tools for the FDF environment. The basic ISPF 
capabilities include a screen-oriented editor; utilities for file 
allocation, copy, display, and code comparison; and both 
foreground and background processing functions. Customization 
has added such features as a file translation utility, a system tape 
generator, specialized print utilities, and transfer functions for 
software moving between the STL and FDF environments. SDE 
also provides access to the PANVALET text management system, to 
the PANEXEC library management system, to Configuration 
Analysis Tool (CAT), to the RSL, and to source code analyzers. 


MEASURES 


Many of the same measures used during detailed design continue to 
be collected and analyzed during the implementation phase. In 
addition, source code generation and the use of configured libraries 
provide the manager with new yardsticks of system growth and 
change. 


Objective Measures 


The following measures are collected during the implementation 
phase: 


e The number of units coded/read/tested versus the number 
identified . 

e Requirements questions and answers, TBDs, and changes 

e Estimates of system size, effort, schedule, and reuse 
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* Staff hours 

* Total CPU hours used to date 

e Source code growth 

¢ Errors and changes by category 


Table 7-1 lists each measure, the frequency with which the data are collected and evaluated, 
and the sources from which the data are obtained. 


Table 7-1. Objective Measures Collected During the Implementation Phase 


a 


== FREQUENCY DATA COLLECTION 
(COLLECT/ANALYZE)} | CONTINUED |] 


Staff hours (total and 
by activity) 


Changes (by category) 


Changes (to source 
files) 


Computer use (CPU 
hours and runs) 


Errors (by category) 
Requirements (changes 


and additions to 
baseline) 


Requirements (TBD 
specifications) 


Requirements 
(questions/answers) 


Estimates of total SLOC 
(new, modified, and 
reused), total units, 
total effort, and 
schedule 


SLOC in controlled 
libraries (cumulative) 


Status (units identified/ 
coded/code-certified/ 
test-certified) 


Developers and 
managers 
(via PRFs) 


Developers (via CRFs) 


Automated tool 


Automated too! 


Developers (via CRFs) 


Managers (via DSFs) 


Managers (via OSFs) 


Managers (via DSFs) 


Managers (via PEFs) 


Automated tool 


Managers (via DSFs) 
(The number of 
completed units is also 
reported by developers 
via COFs and by 
automated tools) 


Weekly/monthly 


By event/monthly 


Weekly/monthly 


Weekly/biweekly 


By event/monthly 


Biweekly/biweekly 


Biweekly/biweekly 


Biweekly/biweekly 


Monthly/monthly 


Weekly/monthly 


Weekly/biweekly (Status data differ from 


those collected during 
design phases) 
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Evaluation Criteria 


The number of units coded, code-certified, and unit-test-certified, 
versus the total number of units to be implemented, are the measures 
of development status collected during the phase. By tracking each 
of these measures on a single graph, SEL managers can see whether 
all activities are progressing smoothly and in parallel. Sudden 
increases or convergences, such as those shown in Figure 7-6, 
should raise a red flag. When the development team is under 
pressure to meet schedules, code reading and unit testing can 
become hurried and superficial. If time is not taken to verify each 
unit properly, the effort needed to complete system testing will be 
increased substantially. 


In the SEL, the growth in the number of units in the project's 
configured library is also tracked against the number of COFs. This 
helps managers ensure that configuration management procedures 
are being followed and that the data needed to track the origin and 
types of system components are being collected. 


Requirements TBDs and changes continue to be tracked during the 
implementation phase. Because designing a system based on best 
guesses can lead to extensive rework, a system should not pass 
CDR with requirements missing. However, if major changes or 
additions to requirements are unavoidable, the design of that portion 
of the system should be postponed and presented in a BDR at a later 
date. One corrective measure for late specifications is to split the 
development effort into two releases, with the late specifications 
included in the second release. 


Analysis: 


development 
status 


requirements 
TBDs and changes 


For most of the 
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implementation phase, code reading and 
unit testing activities followed unit 
coding at a steady rate. However, near 
the end of the phase, nearly three times 
the normal number of units were 
completed in a single week (1). This 
‘miracle finish" was due to short cuts in 


Da code reading and unit testing that were 
: taken in an effort to meet schedules. 


ww, 
a 3 
Result: Project entered the system 


es Sores E testing phase with poor quality 
> Units Read software. To bring the software up to 
> Units Tested standard, the system test phase took 
100% longer than expected. 


IMPLEMENTATION PHASE 


Figure 7-6. Development Profile Example 
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, As implementation progresses, managers 
estimates can obtain more accurate estimates of the 
total number of units and lines of code in the 
system. They can use this data to determine 
whether enough effort has been allocated to 
complete development. 


Section 6 of The Manager's 
Handbook for Software 
Development (Reference 12) 


Managers can compute productivity rates to 
contains additional further refine project estimates and to 
rE ee Ai ahaa compare the pace of implementation with 
cost, and schedule during the that of previous projects. Factors that 
implementation phase. should be considered when measuring 
productivity include the number of lines of 
source code in configured libraries, the 
number of units in the libraries, and the 
number of completed pages of 
documentation per staff hour. 


Staff hours are tracked throughout the phase. If more effort is being 
required to complete a build than was planned, it is likely that the 
remaining builds (and phases) will require proportionally more 
effort as well. After investigating why the deviation has occurred, 
the manager can decide whether to increase staffing or senegal and 
can replan accordingly. 


staff hours 


The profile of computer usage on any project is heavily dependent 
on both the development process and environment. The manager 
must use models of CPU usage from previous, similar projects for 
comparison. In the flight dynamics environment, projects that are 
developing AGSSs show a steep upward trend in CPU usage early 
in the implementation phase. This trend continues during system 
testing, but declines in acceptance testing, when testers conduct 
extensive off-line analysis of numerical results. 


CPU usage 


CPU hours that differ substantially from the local model can be 
caused by insufficient testing or by requirements changes that 
necessitate redesign (Figure 7-7). 


The amount of source code in the project's configured library is a 
key measure of progress during the implementation phase. As with 
CPU usage, the pattern of growth is heavily dependent on the 
development process. On projects with multiple builds, periods of 
sharp growth in configured SLOC will often be separated by periods 
of more moderate growth, when the development team is engaged in 
testing the build. 


source code 
growth 
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Symptom: Computer usage 
zero midway through imple- 
mentation (1). 


Cause: Redesign in response 
to excessive requirements 
changes instead of imple- 
mentation. 


Corrective Action: Replan 
project based on new scope 
of work (2). 


Note: The local model is 
80 10 shown in gray. 


40 0 
WEEKS FROM SRA 


Figure 7-7. Example of CPU Usage — ERBS AGSS 


Managers begin to track change and error data as soon as there are 
units in the configured libraries. These data are usually graphed as 
the cumulative number of changes or errors per thousand SLOC 
over time. 


Developers complete a CRF for each logical change made to the 
software, recording which units were revised as a result of the 
change, the type of change, whether the change was due to error, 
and the effort required. This information is compared with change 
data generated from the configuration management system (€.g., 
CMS) to ensure that the data are consistently reported. 


The rate of software change is a key indicator of project stability. 
Comparative models for change rates should be based on historical 
data from earlier, similar projects. The SEL model, for example, 
reflects a steady, even growth of software changes from the middle 
of the implementation phase through the middle of acceptance 
testing. Exaggerated flat spots in the graph or sudden jumps in the 
change rate should always spur investigation. Excessively high 
rates can result from requirements changes, inadequate design 
specifications, or insufficient unit testing. 


Error rates are generally at their highest level during the 
implementation phase. Error rates in the system test phase should 
be significantly lower, and should show a further decline during 
acceptance testing. The SEL has found that error rates are reduced 
by approximately half in each phase after implementation, and that 
this decline is independent of the actual values involved. Higher 
error rates than expected usually mean that the quality of the 
software has suffered from inadequate effort at an earlier stage, 
although such rates may also be found when the development 
project is exceptionally complex. 
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PRODUCTS 
The key products of the implementation phase are 


* System code and supporting data 

* A set of build test plans and results 
¢ The system test plan 

¢ The analytical test plan 


These products are discussed in the paragraphs that follow. In 
addition, the development team generates a draft of the user's guide, 
while the acceptance test team produces an updated version of the 
acceptance test plan. Since both of these documents are finalized 
during the next phase, they are described in detail in Section 8. 


System Code, Supporting Data, and System Files 


By the end of the last build, the project's configured libraries will 
contain all the source code for the completed system (or release), the 
control and command procedures needed to build and execute the 
system, and all supporting data and system files. 


Included in the supporting files are all input data sets needed for 
testing. Appropriate test data are obtained or generated for each set 
of build tests. By the end of the implementation phase, a full suite 
of input data sets must be ready for use in system testing. If testing 
is to be effective, these test data must be realistic. Test data are 
created using a simulator or data generation tool or by manual data 
entry. Input data for complex calculations are provided to the 
development team with the analytical test plan (see below). 


Build Test Plans 


The use of a formal test plan allows build 
testing to proceed in a logically organized 
manner and facilitates agreement among 
managers and developers as to when the 


Effective testing depends on the timely 
availability of appropriate test data. 


The sonare manda: ment mam mus testing is satisfactorily completed. The 
ensure tha 6 activity of test data 

generation is begun well in advance of development team prepares the test plan 
testing so that neither schedules nor for the first build before the end of the 
tasting quality are:compromised detailed design phase. The test plan for 


because of deficient data. Be 
each subsequent build is prepared before 


the beginning of implementation for the 
build, and highlights of the plan are 
presented at the BDR. 
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Build test plans follow the general outline shown in Figure 7-8. 
Build test plans should always identify a set of build regression tests 
— key tests that can be rerun to ensure that capabilities previously 
provided remain intact after corrections have been made or a new 


build has been delivered. 
System Test Plan 


The system test plan is the formal, detailed 
specification of the procedures to be followed 
while testing the end-to-end functionality of 
the completed system. This test plan follows 
the same general pattern as the build and 
acceptance test plans (Figure 7-8). It is 
reviewed by the system test team and the 
management team prior to the STRR. 


The system test plan must be designed to 
verify that the software complies with all 
requirements and specifications. It should 


concentrate on the user's view of the system . 


and should probe for any weaknesses that 
might not have been uncovered during build 
testing. 


In the Cleanroom methodology, 
tests are statistically selected 
froma hierarchy of possible 


user operations. Build tests are 
scaled-back versions of system tests, 
with input restrictions. Because test 
cases are based on the expected use of 
the system, continuous feedback on the 
reliability of the software is obtained 
with each build test. 


The testing prescribed by the plan should be functional rather than 
structural. In functional testing, the system is treated like a black 
box. Input is supplied and the output of the system is observed. 
The system test plan identifies the functional capabilities specified in 
the requirements and specifications and prescribes a set of input 
values that exercise those functions. These inputs must include 
boundary values and error conditions as well as the main processing 


stream. 


System tests should cover multiple 
operational scenarios, not merely the 
nominal case (e.g., when the 
spacecraft is at a particular attitude 
and orbit position). The test plan 
must include tests designed to ensure 
that interfaces among subsystems are 
thoroughly demonstrated, as well as 
tests that challenge the robustness of 
the system by examining its 
performance under load and its 
response to user or data errors. 
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See Section 8 for 
more information 
on the activites, 
methods, tools, 


Testing tools, such as the 
DEC/Test Manager, can 


help the development team and products of 


to create and organize test 
descriptions and scripts 
efficiently. 


system testing. 
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TEST PLAN OUTLINE 


The recommended outline for build, system, and acceptance test plans is given below. 
Interdependencies among tests should be minimized to allow testing to proceed while failures are 
analyzed and corrected. 


1. Introduction 
a. Brief overview of the system 
b. Document purpose and scope 


2. Test Procedures 

. Test objectives — purpose, type, and level of testing 

. Testing guidelines — test activity assignments (i.e., who builds the executables and who 
conducts the tests), test procedures, checklists/report forms to be used, and configuration 
management procedures 

. Evaluation criteria — guidelines to be used in determining the success or failure of a test 
(e.g., completion without system errors, meets performance requirements, and produces 
expected results) and the scoring system to be followed 

. Error correction and retesting procedures, including discrepancy report forms to be 
completed (see Section 8) 


Test Summary 

a. Environmental prerequisites — external data sets and computer resources required 

b. Table summarizing the system or build tests to be performed 

c. Requirements trace — matrix mapping the requirements and functional specifications to 
one or more test items 


Test Descriptions (Items a to f are repeated for each test) 
a. Test name 

. Purpose of the test — summary of the capabilities to be verified 

. Method — step-by-step procedures for conducting the test 

. Test input f 

. Expected results — description of the expected outcome 

. Actual results (added during the testing phase) — description of the observed results 

in comparison to the expected results 


5. Regression Test Descriptions (Repeat items 4a to 4f for each regression test) 


Figure 7-8. Generalized Test Plan Format and Contents 


System test plans must specify the results that are expected from 
each test. Plans should refer to specific sections of the requirements 
and specifications if these documents already contain expected 
results. Where possible, each test should be defined to minimize its 
dependence on preceding tests, so that the testing process can adapt 
to inevitable, unplanned contingencies. 


The system test plan must also define the set of regression tests that 
will be used to ensure that changes made to software have had no 
unintended side effects. The regression test set should be short 
enough to be actually used when needed, yet should exercise a 


regression 
tests 
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maximum number of critical functions. As a 
rule of thumb, select the key 10 percent of the 
total number of tests as the regression test set. 
: See METHODS AND TOOLS 
in Section 8 for additional 


Analytical Test Plan information on regression 
testing. 

In the flight dynamics environment, an 

additional analytical test plan is generated 

during the implementation phase to assist 

testers in verifying the results of complex 

mathematical and astronomical algorithms. 


cs 


The analytical test plan is produced by 


members of the acceptance test team and is i : 
š if a complete analytical test plan is 


provided to the development team in two available early in the implementation 
phases. Test procedures for verifying phase, the system test plan can be 
i h written to incorporate the analytical tests. 
: computations that are performed at the unit Otherwise, the analytical test plan is 
i level are delivered before the start of the build akan i paralla w e am test 
= . o : 5 pian. In e latter case, e test team 
containing the relevant units. The second must work to efficiently coordinate both 


mnn 


AI PAIL a 


oo kd) 


=æ 
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portion of the analytical test plan, which sets of tests, minimizing the effort spent 
in setup, execution, and analysis. 


contains tests of end-to-end functionality, is 
provided to developers before the start of 
system testing and is executed along with the 
system test plan. 


Analytical test plans are only useful to the development team if input 
data and output results are explicitly specified. Ideally, test data sets 
containing analytically robust, simulated data should be supplied to 
the development team with the plan. The test plan must itemize the 


‘expected, numerical input and output for each test as well as any 


intermediate results that are needed to verify the accuracy of 
calculations. 


BUILD DESIGN REVIEWS 


Reviews are recommended for each build. At BDRs, the 
development team and its managers cover important points of 
information that need to be transmitted before the next phase of 
implementation. Such information includes any changes to the 
system design, the contents of the build, and the build schedule. 


The focus of a BDR is on status and planning. Strategies for han- 


dling TBDs, risk-management plans, and lessons learned from 
previous builds should also be covered. 
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FINISH 


BDR FORMAT 
Presenters — software development team 


Participants 

« Requirements definition team 

« Acceptance test team representatives 
* Quality assurance representatives 

* Customer interfaces 

+ User representatives 

e System capacity/performance analysts 


Attendees must be familiar with the project background, requirements, 
and design. 


Schedule — before implementation of each build of a system or release, 
except the first 


— presentation of changes to the detailed design of the system 
and to the build plan, emphasizing lessons learned in previous builds and 
risk mitigation strategies 


Materials — distributed a minimum of 3 days before the review. 


Figure 7-9. BDR Format 


The formality of the review depends on the size and complexity of 
the project. Large projects may find that a slide presentation and 
hardcopy handouts are necessary. On smaller projects, developers 
may simply meet with analysts, customers, and users around a 
conference table. 


A synopsis of the BDR format is shown in Figure 7-9, and a sug- 
gested outline for the contents of BDR materials is provided in 
Figure 7-10. If a formal presentation is made, the materials 
distributed at the review should be hardcopies of the presentation 
viewgraphs or slides. 


EXIT CRITERIA 


At the end of the final build, the software development manager should 
answer the following questions: 


* Have all components of the system passed each stage of 
verification, inspection, and certification? Are all components 
organized into configured libraries? 


* Have all build test plans been completed? Have all critical 
discrepancies been resolved successfully? 
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+ Has the system test plan been completed and reviewed? Are data 
files and procedures in place for system testing? 


e Are documentation products complete? That is, have all SENs 
been checked and systematically filed in the project library? Is 
the draft of the user's guide ready for evaluation by the system 
test team? 


When the manager can comfortably answer "yes" to each question, 
the implementation phase is complete. 


MATERIALS FOR THE BDR 
. Agenda — outline of review material 


Design changes since CDR or the previous BDR (with justifications) 

a. Revised design diagrams 

b. Changes to system operation; updated operations scenarios/scripts, displays, reports, 
and screens 


Build contents 
a. Requirements to be met in the build 
$ b. Units and data objects to be included in the build 
l c. Unresolved problems in earlier builds to be resolved in this build 


Testing strategy — sequence of build tests, test data, drivers/simulators, etc. 
Changes to remaining builds and releases 


a. Changes in the distribution of requirements among remaining builds/releases 
b. Changes in the distribution of software units and data objects 


Updated software size estimates 


, Milestones and schedules 
a. Schedule for implementing and testing the current build 
b. Schedule for the remaining builds/releases 


Issues, risks, problems, TBD items 
a. Review of any remaining TBDs 
b. Risks associated with the build 


Figure 7-10. BDR Materials 
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SECTION 8 


THE SYSTEM TESTING PHASE 


ENTRY CRITERIA 


* All system code and supporting data 
generated and tested 

* Build test plans successfully executed 

* System test plan completed 

° User. s Mri drafted 


PRODUCTS 


* Tested system code and 
supporting files 
e System test results 
e User's guide 
e System description document 
e Acceptance test plan 
SSO RNa RISER RARE SE HERRERO 


MEASURES 


e System tests planned/executed/passed 

e Discrepancies reported/resolved 

e Staff hours 

e CPU hours 

¢ SLOC in controlled libraries (cumulative) 


« Changes and errors (by category) 

* Requirements Q&As, TBDs, and changes 

e Estimates of system size, effort, 
schedule, and reuse 


EE SSPE OD BO SEE OED ORDO DENG 
BY METHODS AND TOOLS 


e System test plan 

e Regression testing 

e Configuration management 
e Configuration audits 

° Test tools 

* Test logs 

. eye reports 


PE 


EXIT CRITERIA 


« System and analytical test plans 
successfully executed* 
* Acceptance test plan finalized 
* User's guide and system description completed 
e Configuration audits and ATRR conducted 


KEY ACTIVITIES 


System Test Team 
* Prepare for system testing* 
e Execute all items in the system and 
analytical test plans 
* Analyze and report test results 
* Control the testing configuration 
« Evaluate the user's guide 
« Conduct an ATRR 


Development Team 
e Correct discrepancies found during testing 
* Tune system performance 
«Complete system documentation 
* Identify candidates for the RSL 
* Prepare for acceptance testing 


Management Team 
* Reassess schedules, staffing, and resources 
* Ensure the quality and progress of testing 
* Control requirements changes 


* Conduct configuration audits 
* Coordinate the transition to acceptance 


testing 


Acceptance Test Team 
* Finalize the acceptance test plan 
* Prepare for acceptance testing 


* In the Cleanroom methodology, there i is no ; separate system testing phase. All testing 
is conducted by an independent test team in parallel with software development activities. 


Test cases are ponere statistically basea on operaren scenarios. 
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OVERVIEW 


The purpose of the system testing phase is to verify the end-to-end functionality of the 
system in satisfying all requirements and specifications. 


During this phase, the system test team executes the tests specified in the system and 
analytical test plans. The results obtained during test execution are documented, and the 
development team is notified of any discrepancies. Repairs to software are handled by 
members of the development team in accordance with stringent configuration management 
procedures. Corrected software is retested by the system test team, and regression tests are 
executed to ensure that previously tested functions have not been adversely affected. (See 


Figure 8-1.) 


SYSTEM TEST 
PLAN 


PREPARE FOR 
SYSTEM 
TESTING 
81 


EST DATA AND ED 
COMMAND LANGUAGE SYSTEM TEST RESULTS 


USER'S GUIDE 
(DRAFT) EXECUTE / 
RE-EXECUTE SYSTEM TEST 
TEST CASES: RESULTS 
a2 
ANALYZED TEST 
RESULTS 


CONFIGURED SOURCE CODE 


ANALYZE TEST 
RESULTS 
8.3 DISCREPANCY 
REPORTS : 
WORKING SOFTWARE LIBRARIES 


CONFIGURED SOFTWARE LIBRARIES 


CORRECT 
SOFTWARE 
COPY OF ERRORS CORRECTED UNITS 
CONFIGURED a4 
SOFTWARE 


CORRECTED SOURCE CODE 
UPDATE 
CONTROLLED 
LIBRARIES 
as 


NOTE: See KEY ACTIVITIES for more detailed descriptions of the processes in this diagram. 


Figure 8-1. System Testing 
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During the system testing phase, the software development team 
prepares the documentation for the completed system. The user's 
guide is evaluated and an updated version is published by the end of 
the phase. The development team also records the final design of 
the system in a system description document. 


System testing is complete when all tests in both the system test and 
analytical test plans have either executed successfully or have been 
waived at the authorization of the customer. Near the end of the 
phase, the system and its documentation are audited for 
completeness. The system is then demonstrated to the acceptance 
test team and an acceptance test readiness review (ATRR) is held to 
determine whether the system test phase can be concluded and the 
acceptance test phase begun. 


KEY ACTIVITIES 


System tests are planned and executed by a subset of the 
development team that includes the application specialists. In the 
flight dynamics environment, one or more analysts are added to the 
system test team to ensure the mathematical and physical validity of 
the test results. They also learn to operate the software in 
preparation for acceptance testing. 


The senior application specialist is usually 
designated as the leader of the system test 
team. He or she is responsible for ensuring 
that test resources are scheduled and 
coordinated, that the appropriate test 
environment is established, and that the 
other members of the team understand the 
test tools and procedures. During testing, 
this leader directs the actions of the team, 
When reliability requirements ensures that the test plan is followed, 
are particularly stringent, responds to unplanned events, and devises 


system testing may be workarounds for failures that threaten the 
conducted by an independent 


test team. See METHODS progress of testing. 
AND TOOLS for more 
information on independent 


vaificatign and validation The key activities of the system test team, 
(V&V) procedures. the development team, the management 
team, and the acceptance test team are 
summarized in the paragraphs that follow. 
A suggested timeline for the performance of 
these activities is given in Figure 8-2. 
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SYSTEM TEST 
TEAM 


SOFTWARE 
DEVELOPMENT 
TEAM 


ACCEPTANCE 
TEST TEAM 


MANAGEMENT 
TEAM 


STRR ATRR 
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Prepare for system testing 


Execute test cases in the system and analytical test plans y 
Analyze and report test results 


Evaluate the user's guide 


kg 


Conduct regression tests 


Prepare and conduct the ATRR 


Prepare the system description y 


isolate and correct software discrepancies 


Tune system performance 
Update the user's guide 


Identify candidates for the RSL 


v 


Review the acceptance test plan Prepare for acceptance testing 


Participate in the ATRR 


Vv Deliver draft of the acceptance test plan 


Finalize the acceptance test plan 
Prepare for acceptance testing 


Participate in the ATRR 


Record project history data; reassess schedules, staffing, resources 
Ensure progress, quality, and completeness of system testing 


Update SDMP estimates 


Coordinate the transition to acceptance testing 


Conduct configuration audits, 


Figure 8-2. Timeline of Key Activities in the System Testing Phase 
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Activities of the System Test Team 


+ Prepare for system testing. Establish the system test 
environment. System testing generally takes place on the 
development computer rather than the eventual operational 
computer; however, it may be necessary to rehost the system to 
the target computer if there are critical performance 
requirements. Ensure that any test tools that will be used are 
available and operational. 


Ensure that computer resources and operations personnel are 
scheduled and available. Resolve resource conflicts early; they 
can seriously affect schedules, particularly when real-time 
testing is involved. 


Effective testing depends on the timely availability of appropriate 
test data. Collect and use real data for testing whenever 
possible. When real data cannot be obtained, generate and use 
simulated data. Before testing is begun, ensure that all test data 
and command and parameter files (e.g., JCL and NAMELISTs) 
are physically reasonable, reliable, and analytically robust. 
Maintain test data and control files under configuration 
management. 


* Execute all test items in the system and 
analytical test plans, running the 
regression test set each time a replacement 
load module is installed. Follow the 
procedures prescribed in the test plan. 
Keep printed output for each test execution 
and maintain a test log so that events can be 


a cevelcoer ooa net ida oes accurately reconstructed during post-test 
system te is er own > = : x 
Davclonereonthe systemi tast |. analysis. Document any discrepancies 
team should test sections of the between expected and actual test results. 


system implemented by members 
other than themselves. 


The amount of diagnostic output generated 
during testing can either help or hamper 
analysis. Tailor diagnostics and debug 
output to the test and analysis approach so 
that errors can be isolated expeditiously. 


* Analyze and report test results. Test results must be analyzed 
and interpreted to determine if they correspond to those that were 
expected. Where possible, use automated tools in the analysis 
process. Keep good records of analysis procedures and retain 
all relevant materials. 
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Determine whether discrepancies are 
caused by software or by incorrect 
operations. Rerun any tests that failed 


because of errors in data, setup, or The results ot een taning 
procedures as soon as these have been aoa SI as ihe sate 
corrected. When software is implicated, test pian rie CPOE 

p : elow and in Section / tor 
report the problem using discrepancy guidance in reporting test 


mal 


rab 


lod lit 


Wl 


report forms. Discrepancy reports are results. 


reviewed by the management team and 
assigned to members of the development 
team for correction. 


When all system tests are complete, compile a final report 
documenting the results of testing in both detailed and summary 
form. Address any problems encountered and their solutions. 


Control the testing configuration. System testing must yield 
reproducible results. Moreover, when test results do not match 
those that are expected, it must be possible to vary the input 
parameters to find those to which the unexpected results are 
sensitive. During this process, the exact system configuration 
must be kept constant. 


To reduce the possibility of configuration errors, only the project 
librarian should change configured source libraries and build the 
executable image that will be used during system testing. 


Evaluate the user's guide. Employ the user’s guide during test 
preparation and execution. Annotate a copy of the guide, noting 
any errors or discrepancies between the information it contains 
and actual operational conditions. Suggest ways in which the 
clarity and usability of the guide could be improved. 


Conduct an ATRR. Meet with management, customers, and 
members of the acceptance test and development teams to assess 
the results of system testing and the state of preparations for 
acceptance testing. Identify and discuss any outstanding 
problems that may impose technical limits on the testing or may 
affect schedules. The system test and development teams must 
be certain that the system is complete and reliable before it is sent 
to the acceptance test team. 
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Activities of the Development Team 


Wik ie tipi 7 * Correct discrepancies found during 
en comp eting a orm, De . r : : : 
süre:to correctly note tha original testing. Assist the test team in isolating 
source of an error. Changes to code the source of discrepancies between 
may be caused by incorrect require- 
ments and specifications or by expected and actual test results. If the 
design errors. Such changes should error is in the software design, thoroughly 
not be labelled as code errors, even analyze the ramifications of any design 
though coda was revised. : 
changes. Update the affected design 
diagrams and structure charts before 


proceeding with corrections to code. 


Verify all corrections using code reading, 
unit testing, and integration testing. 
Update the SENs for the revised units and 


The contents of the user's fill out a CRF describing the 


guide, system description, Fe er are 
and acceptance test plan are modifications. 


discussed under the : ; 
PRODUCTS heading in this * Tune the performance of the system. 


section. 
Analyze the performance of the system, 


using tools such as the VAX Performance 
and Coverage Analyzer or TSA/PPE (see 
Methods and Tools, Section 7). Locate 
and correct any bottlenecks found. 


* Complete the system documentation. Update the user's 
guide, correcting any errors that were found by the system test 
team and incorporating the team’s recommendations for 
improving the document’s quality. Ensure that the user's guide 
reflects any modifications to software or operational procedures 
that are made as a result of system testing. Deliver the updated 
version of the guide to the acceptance test team at the ATRR. 


Complete the system description. Update the draft begun during 
the implementation phase so that the document reflects the final 
state of the system. By the end of the system testing phase, 
deliver the completed draft to the acceptance test team. 


* Identify candidates for the RSL. If reuse has been a 
determinant in the system design, decide which of the final 


ay software components are sufficiently generalized and robust to 
be candidates for the RSL. Also identify any nongeneric 
components that could be reused in similar systems. Document 


this analysis and forward the candidates for inclusion in the 
RSL. 
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Prepare for acceptance testing. Review the draft of the 
acceptance test plan. Ensure that the plan tests only what is in 
the requirements and specifications document. Any additional 
performance tests or tests that require intermediate results not 
specified in the requirements must be negotiated and approved 
by the management team. 


Work with acceptance testers to obtain the computer resources 
needed for acceptance testing and to prepare the necessary input 
data sets and command procedures (JCL/DCL). If the system 
will be operated in an environment that is different from the 
development environment, rehost the system to the target 
computer. Demonstrate the system to the acceptance test team 
and participate in the ATRR. 


Activities of the Management Team 


Reassess schedules, staffing, and resources. At the 
beginning of system testing, record measures and lessons 
learned from the implementation phase and add this information 
to the draft of the software development history. Use measures 
of effort and schedule expended to date to reestimate costs and 
staffing levels for the remainder of the project. 


Ensure the quality and progress of system testing. 
Coordinate the activities of the system test and development 
teams, and ensure that team members adhere to procedures and 
standards. Place special emphasis on enforcing configuration 
management procedures; the control of software libraries is 
critical during the system and acceptance testing phases. 


On a weekly basis, analyze 
summaries of testing progress and 
examine plots of test discrepancies. 
Ensure that the development team 


aol 


amit Aue 


corrects software errors promptly so The key to success in system 
: testing is the system test plan. 
that testing does not lose momentum. A complete and detailed 
; Assist developers in resolving systern tast plan rasake pa 
7 : precise understanding of the 
z technical problems when necessary. functionality that the tester 


needs to verify, and provides 
Ensure that error corrections are an easy tracking mechanism 

for monitoring weekly testing 
thoroughly tested by the status. 
development team before revised 
software is promoted to controlled 
2 libraries for regression testing. 
z Review all test results and system 


documentation. 
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e Control requirements changes. Although requirements 
changes are not frequent this late in the life cycle, when they do 
occur, the same formal recording mechanisms (i.e., 
requirements question-and-answer forms and specification 
modifications) are used as in preceding life cycle phases. 


Challenge any specification modifications that are received after 
the beginning of system testing. In general, new specification 
modifications that add requirements or enhance the system 
should not be accepted unless they are critical for mission 
support. Moreover, unless mission-critical changes can be 
incorporated with little or no impact to budget and schedule, they 
should be scheduled into a later release or implemented during 
maintenance and operations. 


* Conduct configuration audits. When 
system testing is complete, select one or 
more quality assurance personnel or 
developers to audit the system 
configuration. Determine if test records 


See METHODS AND TOOLS demonstrate that the system meets all 
for the specific procedures to 


be followed in conducting requirements and functional specifications, 
configuration audits. and verify that the system documentation 
completely and accurately describes the 
actual software in controlled libraries. 
Develop action items for the solution of 
any problems found. 


. Coordinate the transition to the acceptance testing phase and 
direct the ATRR. Designate the developers and application 
specialists who will provide support to the acceptance test team 
during the next phase. They will be responsible for setting up 
and running tests with members of the acceptance test team 
present. Supervise demonstrations of the system for the benefit 
of the acceptance test team. 


Schedule the ATRR. Ensure that appropriate representatives 
attend and that the agenda covers any and all issues that could 
affect the success of acceptance testing. Make certain that all 
members of both the acceptance test and development teams 
understand and approve the procedures that will be followed 
during testing. Assign action items resulting from the meeting 
and oversee their resolution. 
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Activities of the Acceptance Test Team 


* Finalize the acceptance test plan. At the start of the system 
test phase, provide the development team with a draft of the 
acceptance test plan. Update and complete the plan during the 
remainder of the phase. 


+ Prepare for acceptance testing. Prepare all test data, control 
language (JCL/DCL), and parameter files needed for testing. 
Generate or request any simulated data that will be needed. 
Verify test data for completeness and accuracy, and place them 
under configuration management. 


Schedule the resources needed for testing, including personnel, 
terminals, and disk space. Optimize resource usage and avoid 
conflicts; careful scheduling is crucial to testing success. 


Attend the system demonstrations conducted by the development 
team. Practice running the system with developer support. 
Participate in the ATRR. 


METHODS AND TOOLS 


The key methods and tools of the system testing phase are 


* The system test plan 

e Regression testing 

« Configuration management 
e Configuration audits 


* Test tools 

* Test logs 

¢ Discrepancy reports 
© IV&V 


The system test plan, which is the primary “tool” used during the 
phase, is a product of the implementation phase and was discussed 
in Section 7. The other methods and tools in this list are elaborated 
in the paragraphs that follow. 


Regression Testing 


Regression testing is the testing that must be performed after 
functional improvements or corrections have been made to the 
system to confirm that the changes have created no unintended side 
effects. Regression testing is an essential element of build and 
acceptance testing as well as of system testing. In the 
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implementation phase, build regression tests are 
run to ensure that a new build has not impaired 
the functioning of previous builds. During 
system and acceptance testing, regression tests 
are conducted by the test team each time the 


For more information on how to load module is replaced. These regression tests 
select a regression test set, see l d b f the full f 

the description of the system are a selected subset of the full set of system or 
test plan in Section 7 and the acceptance tests, and are specified in the test 
discussion of the acceptance plan 


test plan under the PRODUCTS 
heading in this section. 


ED | IE Regression tests also help assure that 

configuration management procedures are 
followed. If regression tests reveal that an 
outdated or untested unit or module was 
included in the test configuration, the 
management team should immediately 
investigate to determine which configuration 
management procedures (described below) 
were violated. 


Configuration Management 


During the system test phase, strict adherence to configuration 
management procedures is essential. Because the software is under 
configuration control at this time, any changes to the code in the 
permanent source code libraries must be made according to 
established procedures and recorded on CRFs. Configuration 
management procedures must ensure that the load modules being 
tested correspond to the code in the project's libraries. 


During system testing, configuration management problems can be 
avoided by following certain basic principles: 


« Limit update access to controlled libraries and restrict rebuilding 
of test configurations to a single configuration manager (usually 
the project librarian). Whenever possible, use automated tools, 
such as CMS control lists, to enforce this restricted access. 


¢ Periodically rebuild the test configuration from the controlled 
source code, eliminating lower-level working libraries, so that 
the system test team can start from an updated, configured 
system on each round of testing. If possible, use a two-level 
library structure. The top level contains the official system test 
executable image built from source code in the controlled library; 
all system testing is performed from this library. The lower- 
level library is used by developers for testing changes when 
system tests have failed. When developers’ changes are 
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promoted into the controlled source code, 


“ou 


= 
= 


the top-level library is rebuilt and the TAILORING NOTE 
lower-level library is emptied. Failed tests eu eis cee ; 
n a projects, managemen 
are then rerun by the test team from the top of a third library, the Ada 
level. compilation library, is critical to 
keeping a controlled system 
F . : configuration. When updated 
*« Restrict the frequency with which new source code is promoted to 
load modules/executable images are controlled libraries, the compila- 
wag tte tion library must be rebuilt 
constructed to minimize the amount of before the new executable 
regression testing that must be conducted. image is constructed. 


A new load module can be built whenever 
a predetermined number of changes to the 
controlled source have been made or can 
be scheduled on a regular basis, e.g., 
every two weeks. 


SENs must also be updated to reflect changes made to source code 
during this phase. The developer obtains the SENs from the project 
librarian before correcting the software and returns them, along with 
the CRF(s), when changes are completed. The project librarian 
checks each returned SEN to ensure that the required checklists and 
listings have been included. 


Configuration Audits 


After system testing is complete, configuration audits are performed 
to confirm that the system meets all requirements and specifications, 
that it is accurately described in the documentation, and that it does 
not include any unauthorized changes. 


In the functional configuration audit (FCA), selected staff members 
spot-check the system test results against the expected results in the 
test plan(s) to determine if the tests were performed properly and 
completely. They also examine all waivers, verifying that any 
uncorrected discrepancies have been reviewed and approved by the 
customer. 


When the FCA is complete, the auditors provide a list of the items 
they examined and their findings to the management team, along 
with their assessment of the readiness of the system for acceptance 
testing. 


In the physical configuration audit (PCA), auditors compare the 
design of the system (as documented in the system description and 
SENs) against listings of the software in configured libraries to 
verify that these descriptions match the actual, tested system. The 
auditors also check the user's guide against the system description to 
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ensure the two documents are consistent. Ina 
report to the management team, they summarize 
their activities, list the items audited, itemize any 
conflicts found, and recommend action items. 


The staff members selected to 


conduct the FCA should not have Test Tools 

implemented or tested the 

system being audited. Many of the automated tools used to facilitate the 

Developers, QA, or CM personnel 3 : : 

may conduct the PCA. implementation of the system continue to be 
employed during the system testing phase. 
These tools — configuration management 


systems, symbolic debuggers, static code 
analyzers, performance analyzers, compilation 
systems, and code-comparison utilities — are 
discussed in Section 7. 


In addition, test management tools (e.g., 
DEC/Test Manager) can provide an online 
mechanism for setting up the test environment, 
The system test plan must for testing interactive applications in batch 
Gescriba the specie: «=. mode, and for regression testing. Use of such a 
procedures to be followed in : 

recording and evaluating tool allows testers to examine test result files 
system tests and while _ interactively and to generate summary reports of 
correcting test discrepancies. aticall 

The contents of the system test runs automatically. 


test plan are discussed under 


PRODUCTS in Section 7. 
ME SSSRA Test Logs 


The test log is an ordered collection of the indi- 
vidual report forms or checklists that are com- 
pleted by testers during actual test execution. 


Use of a standard test report form or checklist is essential to the 
accurate recording of test results. Each report form should identify 
the test, the tester, the load module/executable image being tested, 
and the date and time of testing. The form should provide space to 
summarize the test case and to record whether the test results 
matched those expected. If a discrepancy is found, it is noted on the 
form and described in detail in a discrepancy report (see below). 


Discrepancy Reports 


Testers fill out a discrepancy report form for any errors that they 
uncover that could not be immediately traced to an incorrect test 
setup or operational mistake. Discrepancy reports are also known as 
software problem reports (SPRs), software trouble reports (STRs), 
or software failure reports (SFRs). A sample SFR form is shown in 
Figure 8-3. 
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SOFTWARE FAILURE REPORT 


Originator: Location: Phone: 
SFR#: Date: Time: 
Test ID: Subsystem: 

Load Module: 


Source of Failure: 


Summary (40 characters): _ 


Failure Level: 


Impact, Analysis, and Suggested Resolution: 


Disposition (Circle One): Accept Reject Date Resolution Is Required: 
Assigned to: Date Completed: 


Notes: a a E aaa 


eea o a aaa SS eee 


Figure 8-3. Sample Software Failure Report Form 
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Discrepancy reports are reviewed by the leader of the system test 
team before being passed to the software development team for 
correction. The priority given to a discrepancy report depends on 
the severity level of the failure. One grading system for failures that 
has been used in the flight dynamics environment defines three 
levels: 


* Level 1 (highest severity): A major error occurred and no 
workaround exists. The test cannot be evaluated further. 
Abnormal termination of the program, numerical errors, or 
requirements that are not implemented are considered level 1 
discrepancies. 


* Level 2: A major error occurred but a software workaround 
exists. Abnormal terminations that can be worked around with 
different user input, small errors in final results, or minor 
failures in implementing requirements are classed as level 2 
discrepancies. 


* Level 3 (lowest severity): A cosmetic error was found. An 
incorrect report format or an incorrect description in a display or 
message are classified as cosmetic errors, as are deficiencies that 
make the software difficult to use but that are not in violation of 
the requirements and specifications. If testing schedules are 
tight, the correction of cosmetic errors may be waived at the 
authorization of the customer. 


The status of discrepancy reports must be monitored closely by the 
system test and management teams. A discrepancy report is closed 
at the authorization of the test team leader when the source code has 
been corrected and both the previously-failed test case(s) and the 
regression test set have been executed successfully. 


V&V 


Independent verification and validation (IV&V) is recommended 
whenever high reliability is required in a particular mission-critical 
application, such as manned-flight software. In the flight dynamics 
environment, IV&V implies that system testing is planned and 
conducted by a team of experienced application specialists who are 
independent and distinct from the development team, The system 
test plan that the team develops must contain additional tests 
designed to determine robustness by stressing the system. 


The management team must identify the need for IV&V in the 


SDMP. An additional 5 to 15 percent of total project costs should be 
budgeted to cover the additional effort required. 
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MEASURES 


Objective Measures 


During the system test phase, managers continue to collect and 
analyze the following data: 


¢ Staff hours 

e Total CPU hours used to date 

¢ Source code growth 

¢ Errors and changes by category 

* Requirements questions and answers, TBDs, and changes 
¢ Estimates of system size, effort, schedule, and reuse 


They also begin to monitor measures of testing progress: 


¢ The number of tests executed and the number of tests passed 
versus the number of tests planned 

¢ The number of discrepancies reported versus the number of 
discrepancies resolved 


Table 8-1 lists each of these measures, the frequency with which the 
data are collected and analyzed, and the sources from which the data are 
obtained. Aspects of the evaluation of these measures that are unique 
to the system testing phase are discussed in the paragraphs that follow. 


Evaluation Criteria 


By tracking the number of system tests executed and passed as a 
function of time, the management team gains insight into the 
reliability of the software, the progress of testing, staffing 
weaknesses, and testing quality. Figure 8-4 shows a sample system 
test profile of a project monitored by the SEL. test 
discrepancies 


test status 


The management team also monitors plots of 
the number of discrepancies reported during 
system testing versus the number repaired. 
If the gap between the number of 


discrepancies identified and the number Managers should use a tool to 
resolved does not begin to close after the aa) assist thara in tracking the = 
‘ 7 Ai] progress of system testing. e 
early rounds of testing, the management team tool should provide standardized 
should investigate. One or more application formats for entering and storing 
ays testing status data, and should 
specialists may need to be added to the panbiete clots of dacrapanciss 
development team to speed the correction found and discrepancies 
process. remaining unresolved, asa 
function of time. 
150 
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Table 8-1. Objective Measures Collected During the System Testing Phase 


Developers and 
managers 
(via PRFs) 


Staff hours (total and 
by activity) 


Developers (via CRFs By event/monthly 


Changes (by category) 


Changes (to source Automated tool Weekly/monthly 


files) 


Automated tool 


Computer use (CPU Weekly/biweekly 


hours and runs) 


Developers (via CRFs 


By event/monthly 


Errors (by cafegory) 


Requirements (changes | Managers (via DSFs) Biweekly/biweekly 


and additions to 
baseline) 


Requirements (TBD Managers (via DSFs) 


Biweekly/biweekly 
specifications) - 


Requirements Managers (via DSFs) Biweekly/biweekly 


{questions/answers) 


Estimates of total SLOC | Managers (via PEFs) Monthly/monthly 
(new, modified, and 
reused), total units, 
total effort, and 


schedule 


SLOC in controlled Automated tool Weekly/monthly 


libraries (cumulative) 


Status (tests Managers (via DSFs) Weekly/biweekly 
planned/executed/ 


passed) 


Managers (via DSFs) Weekly/biweekly 


Test discrepancies 
reported/resolved 


Figure 8-5 shows the SEL model for discrepancy status against which current projects are 
compared. The model is generally applicable for any testing phase. 


If the number of discrepancies per line of code is exceptionally high in comparison with 
previous projects, the software has not been adequately tested in earlier phases. Schedule 
and budget allocations should be renegotiated to ensure the quality of the product. 
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—- = = = — — — — Testsplanned= == == : 


w 
Ee 
9 Tests executed — J>- ——== 
H 
Symptom: Testing starts out well, then 
levels off, and finaily continues at a 
lower rate. 
Cause: Midway through the phase, 
testers found they did not have the input 
coefficients needed to test flight 
software. There was a long delay before 
the data became available, and testing 
momentum declined, 
SYSTEM TEST PHASE 
Figure 8-4. EUVEDSIM System Test Profile 
100% If the project's graph of open discrepancy 
reports is above the curve shown in the 
model, the possibie causes are 
a) inadequate staffing to Discrepancies 
correct errors Fixed 
b) very unreliable software 
c) ambiguous or volatile 
requirements 
Discrepancies If the project's graph of open 
Found discrepany reports is below the 
curve shown in the model, the 
probable causes are 
a) reliable, well-designed software 
b} maintainable software (easy to 
fix errors) 
Open discrepancy 
iar 
0% : 
TIME 
Start of Testing Phase End of Testing Phase 


Figure 8-5. SEL Discrepancy Status Model 


Error rates should be significantly lower during system testing than 
they were during build testing. High error rates during this phase 
are an indicator of system unreliability. Abnormally low error rates 
may actually indicate that system testing is inadequate. 
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PRODUCTS 
The key products of the system testing phase are 


* Completed, tested system code and supporting files 
e System test results 

e The updated user's guide 

¢ The system description 

¢ The acceptance test plan 


System Code, Supporting Data, and System Files 


By the end of system testing, the project's configured libraries 
contain the final load module, the source code for the system 
(including changes made to correct discrepancies), the control 
procedures needed to build and execute the system, and all 
supporting data and command files. 


System Test Results 


_ When system testing is complete, test results are published either as 
a separate report or in an update of the system test plan. The latter 
method lets the reader compare actual with expected results more 
easily and eliminates some redundancy in describing test objectives 
(see Figure 7-8). The individual report forms or checklists used 
during testing to log results are collected in a separate volume. 


System test results are reviewed by the test team leader, the 
management team, and the CCB; they are also audited as part of the 
FCA. Approved test results are controlled using configuration 
management procedures. 


User's Guide 


The user's guide contains the information that will be needed by 
those who must set up and operate the system. A recommended 
outline for the user's guide is shown in Figure 8-6. 


During system testing, the draft of the user's guide that was 
produced by the development team during the implementation phase 
is evaluated and updated. The system test team uses the guide 
during test preparation and execution and provides written 
comments back to developers. By the end of system testing, the 
development team publishes an updated, corrected version of the 
document that reflects the state of the system at the completion of 
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USER'S GUIDE 


The development team begins preparation of the user's guide during the implementation phase. 
Items 1 and 2, and portions of item 3, represent updated material from the detailed design 
document, although some rewriting is expected to make it more accessible to the user. A draft is 
completed by the end of the implementation phase and is evaluated during system testing. At 
the beginning of the acceptance test phase, an updated version is supplied to the acceptance test 
team for evaluation. Corrections are incorporated, and a final revision is produced at the end of 
the phase, The suggested contents are as follows: 


1. Introduction 
a. Overview of the system, including purpose and background 
b. Document organization 
c. Discussion and high-level diagrams of system showing hardware interfaces, external 
data interfaces, software architecture, and data flow 


2. Operations overview 
a. Operations scenarios/scripts 
b. Overview and hierarchy of displays, windows, menus, reports, etc. 
c. System performance considerations 


3. Description for each subsystem or major functional capability 
a. Overall subsystem capability 
b. Assumptions about and restrictions to processing in each mode 
c. Discussion and high-level diagrams of subsystems, including interfaces, data flow, and 


communications for each processing mode 

. High-level description of input and output 

. Detailed description of processing keyed to operator-specified input and actions in terms 
of points of control, functions performed, and results obtained (both normal and 
abnormal, i.e., error processing and recovery) 
For interactive subsystems, facsimiles of displays in the order in which they are 
generated 

. Facsimiles of hardcopy output in the order in which it is produced, annotated to show 
what parameters control it 

. List of numbered messages with explanation of system's and user's actions, annotated to 
show the units that issue the message 


4. Requirements for execution 
a. Resources — discussion, high-level diagrams, and tables for system and subsystems 
Hardware 
Data definitions, i.e., data groupings and names 
Peripheral space considerations — data storage and printout 
Memory considerations — program storage, array storage, and data set buffers 
Timing considerations 
{a) CPU time in terms of samples and cycles processed 
(b) VO time in terms of data sets used and type of processing 
l) Wall-clock time in terms of samples and cycles processed 
. Run information — control statements for various processing modes 

Control parameter information — by subsystem, detailed description of all control 

parameters (e.g, NAMELISTs), including name, data type, length, representation, 

function, valid values, default value, units, and relationship to other parameters 


Figure 8-6. User's Guide Contents 
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testing. This version is 
delivered to the acceptance 
test team at the ATRR for use 
during the next phase. 


TAILORING NOTE 


TAILORING NOTE 
Tailor the content of 


the user's guide to 
highlight the key 
information needed 
by users. Keep the 
writing style 
succinct and easily 
understandable. 


System Description 


In the system description, 
briefly discuss the 
error-handling philosophy that 
has been incorporated into the 
software. For Ada systems, the 
discussion should summarize 
the approach used in raising, 
reporting, and recovering from 
exceptions. 


The system description doc- 
ument records the final design 
of the system. It contains 
detailed explanations of the 
internal structure of the 
software and is addressed to 
those who will be responsible 
for enhancing or otherwise 
modifying the system in the 
future. Figure 8-7 gives the 
outline for the system 
description recommended by 
the SEL. 


Acceptance Test Plan 


The acceptance test plan describes the steps that will be taken during 
the acceptance testing phase to demonstrate to the customer that the 
system meets its requirements. The plan details the methods and 
resources that will be used in testing, and specifies the input data, 
procedures, and expected results for each test. In the plan, each test 
is mapped to the requirements and specifications to show which 
requirements it demonstrates. The requirements verified by a 
test items particular test are called test items. 


The acceptance test plan is prepared by members of the acceptance 
test team following the generalized test plan outline shown 
previously (Figure 7-8). To ensure consistency between the 
requirements documents and the acceptance test plan, members of 
the requirements definition team join the acceptance test team to 
begin working on the plan as soon as the initial requirements and 
specifications have been delivered. As TBD requirements are 
resolved and as specification modifications are approved, the draft 
of the acceptance test plan is updated. At the beginning of system 
testing, the completed draft is provided to the development team for 
review. 
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SYSTEM DESCRIPTION 


During the implementation phase, the development team begins work on the system description by updating 
data flow/object diagrams and structure charts from the detailed design. A draft of the document is 
completed during the system testing phase and a final version is produced by the end of acceptance testing. 
The suggested contents are as follows: 


1. Introduction — purpose and background of the project, overall system concepts, and document 
overview 


2. System overview 
. Overview of operations scenarios 
Design drivers (e.g., performance considerations) and their order of importance 
Reuse strategy 
Results of prototyping efforts 
Discussion and high-level diagrams of the selected system design, showing hardware interfaces, 
external data interfaces, interconnections among subsystems, and data flow 
Traceability matrix of major components against requirements and functional specifications 
Error-handling strategy 


Description of each subsystem or major functional breakdown 
Overall subsystem capability 
Assumptions about and restrictions to processing in each mode 
Discussion and high-level diagrams of subsystem, including interfaces, data flow, and 
communications for each processing mode 
High-level description of input and output 
Structure charts or object-oriented diagrams expanded to the subprogram level, showing interfaces, 
data flow, interactive control, interactive input and output, and hardcopy output 


Requirements for creation 

a. Resources — discussion, high-level diagrams, and tables for system and subsystems 
Hardware 
Support data sets 
Peripheral space considerations — source code storage, scratch space, and printout 
Memory considerations — program generation storage and data set buffers 
Timing considerations 
{a) CPU time in terms of compile, build, and execute benchmark test 
(b) 1/0 time in terms of the steps to create the system 

b. Creation information — control statements for various steps 

c. Program structure information — control statements for overlaying or loading 


Detailed description of input and output by step — source code libraries for system and 
subsystems, object code libraries, execution code libraries, and support libraries 


Internal storage requirements — description of arrays, their size, their data capacity in all 
processing modes, and implied limitations of processing 


Data interfaces for each internal and external interface 

a. Description, including name, function, frequency, coordinates, units, computer type, length, and 
representation 

b. Format — organization (e.g., indexed), transfer medium (e.g., disk), layout of frames 
(samples, records, blocks, and/or transmissions), and storage requirements 


Description of COMMON blocks, including locations of any hard-coded physical constants 


Prologs/package specifications and PDL for each unit (separate volume) 


Alphabetical list of subroutines from support data sets, including a description of each 
subroutine's function and a reference to the support data set from which it comes 


Figure 8-7. System Description Contents 
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Acceptance tests must be designed to verify that the system meets 
existing, documented requirements. All tests for robustness and 

> performance must be based on stated requirements. The plan must 
explicitly specify the results that are expected from each test and any 
debug data or intermediate products that testers will require in order 
to evaluate the software. 


The basic subset of tests that are to be run during regression testing 
are also identified in the test plan. The regression test suite should 
include those tests that verify the largest number of critical 
requirements in a single run, i.e., those that are the most 
comprehensive, yet efficient, tests to execute. 


In general, the acceptance test plan should be constructed so that 

tests can be conducted independently. Multiple tests may be 
= performed simultaneously, and testing should be able to continue 

after a problem is found. Tests must also be repeatable; if two 

different testers execute the same test case, the results should be 
= consistent. 


ACCEPTANCE TEST READINESS REVIEW 


ity 


When the system is ready to be given to the acceptance test team, a 
formal "hand-over" meeting of developers and testers is held. The 
purpose of this acceptance test readiness review (ATRR) is to 
identify known problems, to establish the ground rules for testing, 
and to assist the acceptance test team in setting up the testing 
environment. 


\ 


The ATRR is an opportunity to finalize 
> procedural issues and to reach agreement on 
the disposition of any unresolved problems. 
To facilitate this, the development team should 
conduct demonstrations of the system for the 


The formality of the ATRR should 
be tailored to the project. On 
large projects with many inter- 
facing groups, the ATRR should 
be held as a formal review with 


hardcopy materials and slide/ 
viewgraph presentations. On 
small projects, the ATRR may be 
an informal meeting held around 
a conference table. 


acceptance test team prior to the meeting. At 
the meeting, the discussion should cover the 
status of system tests, specification 
modifications, system test discrepancies, 
waivers, and the results of configuration 
audits. 


The ATRR format and schedule are shown in 


Figure 8-8. An outline of recommended 
materials is provided in Figure 8-9. 
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ATRR FORMAT 


Presenters 

e System test team 

« Acceptance test team 

« Software development team 


Other participants 

e Quality assurance representatives 

e Customer interfaces 

e User representatives 

« System capacity/performance analysts 


Attendees must be familiar with the project background, requirements, 
and design. 


Schedule — after completion of system testing and before the beginning 
of the acceptance testing phase 


Agenda — establishment of ground rules and procedures for acceptance 
testing, identification of known problems, and discussion of the test 
environment 


Materials — distributed a minimum of 3 days before the review. 


Figure 8-8. ATRR Format 


MATERIALS FOR THE ATRR 
Introduction — outline of ATRR materials, purpose of the review, and system overview 


System test results 

a. Summary of test results 

b. Status of discrepancies and waivers 
c. Unresolved issues 


Acceptance testing overview — summary of testing approach, including any major 
changes to the acceptance test plan 
a. Organizational responsibilities during the acceptance testing phase 


. Test configuration and environment, including facilities and computer resources 
Configuration management procedures 

. Test tools, test data, and drivers/simulators 

. Test reporting, analysis, and discrepancy-handling procedures 
Test milestones and schedules 


Test readiness assessment 
. Results of configuration audits 
. System readiness 
Staff training and preparedness 
. Status of the acceptance test plan 
Action items and schedule for their completion 


Figure 8-9. ATRR Materials 
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EXIT CRITERIA 


To determine whether the system and staff are ready to begin the 
acceptance testing phase, the management team should ask the 
following questions: 


e Have the system and analytical test plans been executed 
successfully? Have all unresolved discrepancies either been 
eliminated as requirements or postponed to a later release ina 
formal waiver? 


e Have the FCA and PCA been conducted? Have all resulting 
action items been completed? 


e Are the user's guide and system description accurate, complete, 
and clear? Do they reflect the state of the completed system? 


e Has the acceptance test plan been finalized? 


When the management team can answer "Yes" to each question, the 
system testing phase can be concluded. 
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ACCEPTANCE 
TESTING 


SECTION 9 


THE ACCEPTANCE TESTING PHASE 


RVR 


“con Sena = Bes atasnsSes RoR 
"PHASE HIGHLIGHTS) 


A TET ENTE TY 


ENT 


e System and analytical test plans 
successfully executed 
* Acceptance test plan finalized 


* System delivery tape 

«Finalized user's guide and 
system description 

e Acceptance test results 

* Software development history 


MEASURES 


° Staff hours 

+ CPU hours 

e SLOC in controlled libraries (cumulative) 

+ Changes and errors (by category} 

* Requirements Q&As, TBDs, and changes 

* Estimates of system size, effort, 
schedule, and reuse 

» Acceptance test items planned/ 
executed/passed 

« Discrepancies reported/resolved 

* SEL project completion statistics 
SIS RS 


« Acceptance test plan 

« Regression testing 

e Configuration management 

e Test evaluation and 
error-correction procedures 

« Discrepancy reports, test logs, and 
test management tools 

SRS SICA ER S 


ARR 


RN 


EXIT CRITERIA 


* Acceptance test plan successfully executed 


* User's guide and system description 
finalized 


KEY ACTIVITIES 


Acceptance Test Team 
* Complete the preparations for acceptance 
testing q 
« Execute the tests items in the acceptance 
test plan 
+ Analyze and report test results 


e Evaluate the user's guide 
* Formally accept the system 


Development Team 
* Support the acceptance test team 
«Correct discrepancies found during testing 
* Train the acceptance test team to operate 
the system 
* Refine system documentation 
« Deliver the completed system 


Management Team 
* Allocate team responsibilities and finalize 


-testing procedures 

* Ensure the quality and progress of testing 
* Ensure cooperation among teams 

+ Prepare the software development history 


arena ae 


In the Cleanroom methodology, acceptance testing is complete when the mean time 
between failure (MTBF) of statistically generated tests exceeds a pre-determined 


threshold value. 


PRECEDING PAGE BLANK NOT FILMED 
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OVERVIEW 


The purpose of the acceptance testing phase is to demonstrate that 
the system meets its requirements in the operational environment. 


The acceptance testing phase begins after the successful conclusion 
of an ATRR. The testing conducted during this phase is performed 
under the supervision of an independent acceptance test team (on 
behalf of the customer) and follows the procedures specified in the 
acceptance test plan. The development team provides training and 
test execution and analysis support to the test team. They also 
correct any errors uncovered during testing and update the system 
documentation to reflect software changes and corrections. 


Acceptance testing is complete when each test item specified in the 
acceptance test plan has either been successfully completed or has 
been waived at the authorization of the customer, and the system has 
been formally accepted. The phase is concluded when the system 
delivery tape and final system documentation have been delivered to 
the customer for operational use. 


Figure 9-1 is a flow diagram that shows the process of acceptance 
testing. 


KEY ACTIVITIES 


During this phase, the acceptance test team and the development 
team work closely together to set up and execute the tests specified 
in the acceptance test plan. 


The acceptance test team is composed of the analysts who will use 
the system and several members of the team that prepared the 
requirements and specifications document. The acceptance test 
team supplies test data, executes the tests, evaluates the test results, 
and, when acceptance criteria are met, formally accepts the system. 


Members of the development team correct any discrepancies arising 
from the software and finalize the system documentation. Often, the 
development team sets up and executes the first round of acceptance 
tests. This provides training for the acceptance test team, which 
then assumes responsibility for test execution during the subsequent 
rounds of acceptance testing. In any case, one member of the 
development team is always present during each testing session to 
offer operational and analysis support. 
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ACCEPTANCE TEST 


PREPARE FOR 
ACCEPTANCE 


at 


TEST DATA AND 


a 
COMMAND LANGUAGE ACCEPTANCE TEST REPORT 


USER'S GUIDE AND 
SYSTEM DESCRIPTION EXECUTE/ 
RE-EXECUTE ACCEPTANCE 
TESTCASES / ~  TESTRESULTS 
0.2 ANALYZED TEST RESULTS 
RESULTS 


GURED SOURCE CODE 


ANALYZE TEST 
RESULTS 
1 ` CORRECTED 
DISCREPANCY UNITS 
REPORTS 


CONFIGURED SOFTWARE LISRARIES __, COPY OF WORKING SOFTWARE LIBRARIES 


(OPERATIONAL ENVIRONMENT) CONFIGURED — P | 
SOFTWARE 
CORRECT COARECTED 


SOFTWARE UNITS 
EARORS 
34 
USER'S GUIDE AND 
SYSTEM DESCRIPTION 
——— 


CORRECTED SOURCE CODE UPDATE 
CONTROLLED 


UBSRARIES AND UPOATED 


oe DOCUMENTATION 


FINAL SYSTEM CONFIGURATION 


FINAL SYSTEM CONFIGURATION 


——— ÁÁ 
OPERATIONAL SOFTWARE LIBRARIES@— 
NOTE: See KEY ACTIVITIES for more detailed SUPPORTING FILES, AND 


descriptions of the processes in this diagram. DOCUMENTATION 


Figure 9-1. Acceptance Testing 


Some acceptance testing activities, such as test set up and execution, generation of test data, 
and formal discrepancy reporting, can be performed by either the development team or by 
the acceptance test team. At the ATRR, the assignment of each of these activities to a 
particular team is negotiated by members of the management team. The management team, 
which includes managers of both the software development and acceptance test teams, 


bases these assignments on the specific needs and drivers of the project. 
The key technical and managerial activities of the acceptance test phase are outlined in the 


following paragraphs. A recommended timeline for the execution of these activities is 
shown in Figure 9-2. 
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Complete preparations for acceptance testing 


ACCEPTANCE 
TEST TEAM Generate test data 


Evaluate the user's guide _ 


Execute acce ptance tests 


-y 
ee ns 
Evaluate tests vy 


Report final test results 


—7 

Prepare for acceptance testing 
SOFTWARE 
DEVELOPMENT Provid g 
TEAM rovide acceptance testing support 

Correct errors 
ct erro - y 
Deliver new load modules 
Update user's guide and system description 
-7 
Deliver system 

—F 

Finalize testing procedures and assign team duties 

any. 
MANAGEMENT Record project history data 
TEAM 


Ensure progress, quality, and completeness of acceptance testing 


Coordinate testing activities and manage priorities 


a 


Quality assure final products 


Complete software development history 


TIME a 


Figure 9-2. Timeline of Key Activities in the Acceptance Testing Phase 
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Section 9 - Acceptance Testing 


Activities of the Acceptance Test Team 


- Complete the preparations for acceptance testing. Establish 
the acceptance test environment. Acceptance testing, unlike 
system testing, always takes place on the computer targeted for 
operational use; therefore, it is critically important to schedule 
computer resources well in advance. 


Finish generating the simulated data to be used in testing and 
allocate the necessary data sets. Quality check the simulated data 
to ensure that the required test data have been produced. Bear in 
mind that generating simulated data requires intensive effort and 
that, because of the complexity of flight dynamics test data, 
simulated data must often be generated several times before the 
desired data set is achieved. Plan ahead for adequate time and 
resources. 


Refer to the user's guide to determine the parameters needed to 
set up the test cases. Work with the development team to set up 
the tests accurately and to assure the quality of test preparations. 
Check the control-language procedures (JCL/DCL) for executing 
tests. 


Obtain the expected results from the acceptance test plan and 
review them for completeness. Calculate any expected results 
that are missing. 


* Execute the test items in the acceptance test plan. Early in 
the phase, attempt to execute all tests in the order given in the 
acceptance test plan, executing each item of each test in 
sequence. This "shake-down” ensures that any major problems 
are found early, when they can be most easily corrected. 


If a basic misunderstanding of a requirement is found during 
initial test runs, such that further testing would be unproductive, 
suspend testing until a new load module can be produced. 
Negotiate with the development team to determine a schedule for 
the delivery of a corrected load module and the resumption of 
testing. 


Control the testing configuration. To relate test results to a 
particular load module unambiguously, run the entire set of tests 
designated for a round of testing on a specific load module 
before introducing any corrections. Continue testing until the 
number of errors encountered make further testing unproductive. 
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Introduce new load modules according to the test configuration 
management procedures. When a new, corrected load module is 
received, first rerun those tests that previously failed because of 
software errors. If those tests succeed, proceed with regression 
testing. 


Execute the regression tests to demonstrate reproducible results 
and ensure that corrections have not affected other parts of the 
system. The basic subset of tests to be run during regression 
testing, as identified in the acceptance test plan, includes those 
that are the most comprehensive yet efficient tests to execute. 
The regression test suite may be modified to expedite testing or 
to provide different coverage as testers gain more experience 
with the system. 


During each test session, review test results with the developers 
who are supporting the tests to ensure that the test was executed 
properly. At the end of the test session, produce a preliminary 
report listing the test cases that were executed and recording 
these initial observations. 


Analyze and report test results. Evaluate test results within 
one week of execution. Wherever possible, use automated tools 
in the analysis process. Record analysis procedures and keep all 
relevant materials. Remember that records and reports must give 
complete accounts of the procedures that were followed; if a test 
cannot be evaluated, note the fact and report the reasons for it. 


Compare test results with expected results and document all 
errors and discrepancies found, regardless of how minor they 
appear or whether they will be corrected. 


Prepare a detailed evaluation report for each test. This report 
should include a test checklist and a categorization of the results 
for each item of each test. Specific test evaluation procedures, 
including a five-level test categorization scheme, are discussed 
under Methods and Tools. 


Near the end of the phase, when all acceptance tests have been 
completed, prepare a final report documenting the results of 
testing. Include the final detailed report on each test, and 
provide an overall summary that gives an overview of the testing 
process and records how many test items passed during each 
round of testing. 
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Section 9 - Acceptance Testing 


¢ Evaluate the user's guide. During acceptance testing, evaluate 
the user's guide for accuracy, clarity, usability, and 
completeness. Provide this feedback to the development team so 
that they can update the document and issue a finalized version 
by the end of acceptance testing. 


* Formally accept the system. When the system meets all 
acceptance criteria, except those waived by the customer, 
prepare a formal memorandum declaring the system accepted. 


Activities of the Development Team 


* Support the acceptance test team. Assist the acceptance test 
team in setting up and executing tests. At least one developer 
should be available during each test session to answer questions, 
to provide guidance in executing the system, and to review initial 
test results. 


h 


"osi 


NOTE 


The degree to which the 
development team assists the 
acceptance test team in setting up 
and executing tests, generating test 


data, or preparing discrepancy 
reports varies with the project. 
Management defines each team's 
duties at the start of the phase and 
documents them in the acceptance 
test procedures. 


No specification modifications 
that add requirements or 
enhancements can be accepted 


during acceptance testing. 
Instead, these should be 
carefully considered for 
implementation during the 
maintenance phase. 


Work with the acceptance test team to isolate 
the sources of discrepancies between 
expected and actual test results. Classify 
failed test items according to their causes, 
showing which items failed because of the 
same software discrepancies and which items 
could not be evaluated because a preceding 
test failed. Help the testers prepare formal 
discrepancy reports; typically, because a 
single factor may cause more than one 
failure, there are fewer discrepancy reports 
than failed test items. 


Correct discrepancies found during 
testing. Correct those software errors that 
occur because of faults in the design or 
code. Errors in the specifications are 
corrected as negotiated by the acceptance test 
manager, the development manager, and the 
customer. Because the requirements and 
specifications are under configuration 
control, the CCB must also approve any 
corrections to specifications. Incorporate 
software updates that result from approved 
corrections into a future load module. 
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Use detailed test reports, test output, and the specifications to 
isolate the source of each error in the software. If an error is 
uncovered in the software design, the effect of the repair on 
costs and schedules may be significant. Look for a workaround 
and notify management immediately so that the impact and 
priority of the error can be determined. 


Verify all corrections, using code reading, unit testing, and 
integration testing. Update the SENs for the revised units and 
fill out a CRF describing each correction. Strictly follow the 
same configuration management procedures for revised units as 
were used during system testing. To reduce the possibility of 
configuration errors, only the project librarian should change 
configured source libraries and build the executable image. 


Deliveries of new acceptance test load modules should include a 
memorandum detailing which discrepancy reports have been 
resolved and which are still outstanding. 


Train the acceptance test team to 
operate the system. Ensure that 
members of the acceptance test team 
steadily gain expertise in running the 
system. Training may be completed at 


- minha, Goud mom ain 


gloom. 


Alii. 


the beginning of acceptance testing or in the flight dynamics environment, 

spread out over the phase. The schedule training ta limited so'executing:or 
PRA operating the system. The 

of training depends on when the development team does not train 


Ai mi 


acceptance test team is to assume full the operations team on how to use 
the system to support the mission. 


responsibility for executing the tests. 


Refine system documentation. Finalize 
the user's guide and the system 
description documents to reflect recent 
software changes and user comments. 


Deliver the completed system. After the system is accepted, 
prepare the final system tape. Verify its correctness by loading it 
on the operational computer, building the system from it, and 
using the test data on the tape to execute the regression test set 
specified in the acceptance test plan. 


Activities of the Management Team 
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Allocate team responsibilities and finalize testing 
procedures. Negotiate to determine the appropriate team 
structure, division of duties, testing procedures, and 
contingency plans for the project. 
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Section 9 - Acceptance Testing 


Define a realistic testing process that can be used on this project 
and adhere to it. Tailor the standard process to address drivers 
specific to the project, such as system quality, schedule 
constraints, and available resources. 


Define the roles and responsibilities of each team clearly. 
Specify which teams are to generate simulated data, set up test 
cases, execute tests, and prepare formal discrepancy reports. 
Where teams share responsibility, clearly define the specific 
activities that each is expected to perform. 


Stipulate contingency plans. Specify what will happen if a 
major error is encountered. Define the conditions that warrant 
suspension of testing until a correction is implemented. Define 
when and how testing should circumvent a problem. 


Ensure the quality and progress of testing. Ensure that team 
members adhere to procedures and standards. Analyze 
summaries of testing progress and examine plots of test 
discrepancies weekly. Use measures of effort and schedule 
expended to date to reestimate costs and staffing levels for the 
remainder of the project. 


Place special emphasis on enforcing configuration management 
procedures. The control of software libraries and test 
configurations is critical during the acceptance testing phase. 


Ensure cooperation among teams. Coordinate the activities of 
the acceptance test and the development teams. Ensure that the 
acceptance test team evaluates tests quickly and that the 
development team corrects software errors promptly so that the 
pace of testing does not decline. Assist in resolving technical 
problems when necessary. 


Ensure that error corrections are thoroughly 
tested by the development team before 
revised units are promoted to controlled 
libraries and delivered for retesting. 
Review all test results and system 


The contents of the software documentation. 
development history are 
outlined under the PRODUCTS * Prepare the software development 


heading in this section. 


| 


history. At the beginning of acceptance 
testing, record measures and lessons 
learned from the system testing phase and 
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add this information to the software development history. 


Complete the phase-by-phase history at the end of acceptance 
testing and prepare the remainder of the report. Summarize the 
project as a whole, emphasizing those lessons learned that may 
be useful to future projects and process-improvement efforts. 
Include summary charts of key project data from the SEL 
database. 


METHODS AND TOOLS 
The key methods and tools of the acceptance testing phase are 


* The acceptance test plan 

* Regression testing 

* Configuration management 

¢ Test evaluation and error-correction procedures 
« Discrepancy reports 

* Test logs 

* Test management tools 


The final acceptance test plan, which describes the procedures to be 
followed in recording and evaluating acceptance tests and while 
correcting test discrepancies, is a product of the system testing phase 
and was discussed in Section 8. 


Many of the automated tools used in the acceptance testing phase 
were introduced to facilitate the implementation of the system. 
These tools — configuration management systems, symbolic 
debuggers, static code analyzers, performance analyzers, 
compilation systems, and code-comparison utilities — are discussed 
in Section 7. 


Test logs, discrepancy reports, test management tools, and 
regression testing procedures are the same as those used for system 
testing; they are discussed in detail in Section 8. 


The other methods and tools in this list are elaborated in the 
paragraphs that follow. 


Configuration Management 
During the acceptance test phase, as in the other test phases, strict 
adherence to configuration management procedures is essential. 


Any changes to the code in the permanent source code libraries must 
be made according to established procedures and recorded on CRFs. 
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Section 9 - Acceptance Testing 


Configuration management procedures must ensure that the load 
modules being tested correspond to the code in the project's 
libraries. 


Developers should follow the guidelines given in Section 8 for 
managing configured software as they repair discrepancies, test the . 
corrections, and generate new load modules for acceptance testing. 


Acceptance testers must be able to relate test results to particular load 
modules clearly. Therefore, the load modules must be under strict 
configuration control, and, at this point in the life cycle, the entire 
set of tests designated for a round of testing should be run on a 
specific load module before any corrections are introduced. 
Similarly, changes in the test setup or input data must be formally 
documented and controlled, and all tests should be labeled for the 
module and environment in which they were run. 


Test Evaluation and Error-Correction Procedures 


The acceptance test team is responsible for evaluating each 
acceptance test and forwarding the results to the development team. 


Each test consists of a number of items to be checked. The results 
of analyzing each test item are categorized according to the following 
five-level evaluation scheme: 


* Level 1: Irem cannot be evaluated or evaluation is incomplete. 
This is normally the result of an incorrect test setup or an 
unsatisfied dependency. If this category is checked, the reason 
should be stated. If the test could not be evaluated because the 
test was inadequately specified, the test plan needs to be 
corrected. 


* Level 2: Item does not pass, and an operational workaround for 
the problem does not exist. 


* Level 3: Item does not pass, but an operational workaround for 
the problem exists. 


* Level 4: Only cosmetic errors were found in evaluating the item. 
* Level 5: No problems were found. 

Typically, items categorized as Level 1 or 2 are considered to be 
high priority and should be addressed before items at Levels 3 and 
4, which are lower priority. The intention is to document all errors 


and discrepancies found, regardless of their severity or whether they 
will ultimately be corrected. 
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Tests and test items are evaluated according to these criteria: 


e All runs must execute correctly without encountering abnormal 
terminations or any other runtime errors. 


e Numerical results must be within acceptable limits of the 
expected values. 


¢ Tabular output must meet the criteria outlined in the 
specifications, with proper numerical units and clear descriptions 
of parameters. 


The acceptance test team documents test results using two kinds of 
report: 


¢ Preliminary reports, prepared at the end of each test session, 
provide a rapid assessment of how the software is working and 
early indications of any major problems with the system. They 
are prepared by the acceptance test team on the basis of a "quick- 
look" evaluation of the executions. 


+ Detailed reports, prepared within a week of test execution, 
describe problems but do not identify their sources in the code. 
They are prepared by the acceptance test team by analyzing 
results obtained during the test sessions, using hand calculations 
and detailed comparisons with the expected results given in the 
acceptance test plan. 


The testers and developers then work together to identify the 
software problems that caused the erroneous test results directly or 
indirectly. These software problems are documented and tracked 
through formal discrepancy reports, which are then prioritized and 
scheduled for correction. 


The development team then corrects errors found in the software 
subject to the following guidelines: 


e Software errors that occur because of a violation of the 
specifications are corrected, and the corrections are incorporated 
into an updated load module for retesting. 


e Corrections resulting from errors in the specifications or 
analytical errors require modifications to the specifications. The 
customer and the CCB must approve such modifications before 
the necessary software updates are incorporated into an updated 
load module. 
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Objective Measures 


During the acceptance testing phase, managers continue to collect 
and analyze the following data: 


e Total CPU hours used to date 

e Source code growth 

¢ Errors and changes by category 

+ Requirements questions and answers, TBDs, and changes 
* Estimates of system size, effort, schedule, and reuse 

* Test items planned, executed, and passed 

¢ Discrepancies reported and resolved 


A complete description of the 
PCSF and SEF can be found 
in Data Collection Procedures 


for the SEL Database 
(Reference 19). 


The source of these measures and the frequency 
with which the data are collected and analyzed are 
shown in Table 9-1. 


At the completion of each project, SEL personnel 
prepare a project completion statistics form 
(PCSF). The management team verifies the data 
on the PCSF, and fills out a subjective evaluation 


form (SEF). The data thus gathered helps build 


an accurate and comprehensive understanding of 
the software engineering process. Combined 
with the software development history report, the 
SEF supplies essential subjective information for 
interpreting project data. 


Evaluation Criteria 


The management team can gauge the reliability of the software and 
the progress of testing by plotting the number of acceptance test 
items executed and passed in each round of testing as a function of 
test status time. An exceptionally high pass rate may mean that the software is 
unusually reliable, but it may also indicate that testing is too 
superficial. The management team will need to investigate to 
determine which is the case. If the rate of test execution is lower 
than expected, additional experienced staff may be needed on the test 


team. 
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Table 9-1. Objective Measures Collected During the Acceptance Testing Phase 


Staff hours (total and 
by activity) 
Changes (by category) 


Changes (to source 
files) 


Computer use (CPU 
hours and runs) 


Errors (by category) 


Requirements (changes 


and additions to 
baseline) 


Requirements (TBD 
specifications) 


Requirements 
(Questions/answers) 


Estimates of total SLOC 


(new, modified, and 
reused), total units, 
total effort , and 
schedule 


SLOC in controlled 
libraries (cumulative) 


Status ( test items 
planned/executed/ 
passed) 


Test discrepancies 
reported/resolved 
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Developers and 
managers 
(via PRFs) 


Developers (via CRFs 


Automated tool 
Automated tool 
Developers (via CRFs 
Managers 

(via DSFs) 

Managers 

(via DSFs) 

Managers (via DSFs) 
Managers 

(via PEFs) 


Automated tool 


Managers (via DSFs) 


Managers (via DSFs) 


FREQUENCY DATA COLLECTION 
(COLECTIANALYZ=_ | CONTINUED [BEGUN | 


Weekly/monthly 


By event/monthly 


Weekly/monthly 
Weekly/biweekly 


By event/monthly 


Biweekly/biweekly 


Biweekly/biweekly 
Biweekiy/biweekly 


Monthlysmonthly 


Note: Test status is 
plotted separately for 
the system testing and 
the acceptance testing 
phases. Test 
discrepancies are also 
plotted separately for 
each phase. 
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test 
discrepancies 


error rates 


The management team also monitors plots of the number of 
discrepancies reported versus the number repaired (see Measures in 
Section 8). If the gap between the number of discrepancies 
identified and the number resolved does not begin to close after the 
early rounds of acceptance testing, more staff may be needed on the 
development team to speed the correction process. 


The trend of the error rates during the acceptance testing phase is 
one indication of system quality and the adequacy of testing. Based 
on SEL projects from the late 1980s and early 1990s, the SEL 
model predicts a cumulative error rate of 4.5 errors per KSLOC by 
the end of acceptance testing. Typically, the SEL expects about 2.6 
errors per KSLOC in the implementation phase, 1.3 in system 
testing, and 0.65 in acceptance testing; i.e., error detection rates 
decrease by 50 percent from phase to phase. 


An error rate above model bounds often indicates low reliability or 
misinterpreted requirements, while lower error rates indicate either 
high reliability or inadequate testing. In either case, the management 
team must investigate further to understand the situation fully. 
Figure 9-3 shows an example of an error-rate profile on a high- 
quality project monitored by the SEL. 


Symptom: Lower error rate 
and lower error detection 
rate. 


TATION ST TEST 


Cause: Early indication of 
high ‘quality. Smooth pro- 
gress observed in uncovering 
errors, even between phases 
(well-planned testing). 


Result: Proved to be one of 
the highest-quality systems 
produced in this environment. 


TIME 


Figure 9-3. Sample Error-Rate Profile, VARS AGSS 


175 


Section 9 - Acceptance Testin 


PRODUCTS 
The key products of the acceptance testing phase are 


System delivery tape 

Finalized user's guide and system description 
Acceptance test results 

Software development history 


System Delivery Tape 


At the end of acceptance testing, a system delivery tape is produced 
that contains the accepted system code and the supporting files from 
the project's configured libraries. The system tape holds all of the 
information required to build and execute the system, including the 
following: 


¢ The final load module 

* The source code for the system 

* The control procedures needed to build and execute the system 
¢ All supporting data and command files 

* Test data for regression tests 


Two copies of the system tape are delivered to the customer. 
Finalized User's Guide and System Description 


The user's guide is updated to clarify any passages that the testers 
found confusing and to reflect any changes to the system that were 
made during the acceptance testing phase. The system description is 
also updated; system changes are incorporated into the document 
and any discrepancies found during the configuration audits are 
corrected. Final versions of both documents are delivered at the end 
of the project. 


Acceptance Test Results 


When acceptance testing is complete, test results are published either 
as a separate report or in an update of the acceptance test plan. (The 
latter method allows the reader to easily compare actual with 
expected results and eliminates some redundancy in describing test 
objectives; see Figure 7-8.) The individual report forms or 
checklists used during testing to log results are collected in a 
separate volume. 
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Section 9 - Acceptance Testing 


FINISH 


Software Development History 


During the acceptance testing phase, the management team finalizes 
the software development history for the project. This report, which 
should be delivered no later than one month after system acceptance, 
summarizes the development process and evaluates the technical and 
managerial aspects of the project from a software engineering point 
of view. The purpose of the report is to help software managers 
familiarize themselves with successful and unsuccessful practices 
and to give them a basis for improving the development process and 
product. The format and contents of the software development 
history are shown in Figure 9-4. 


EXIT CRITERIA 


To determine whether the system is ready for delivery, the 
management team should ask the following questions: 


+ Was acceptance testing successfully completed? 


* Is all documentation ready for delivery? Does the user's guide 
clearly and completely describe how to execute the software in 
its operational environment? 


* Has the acceptance test team, on behalf of the customer, 
formally accepted the software? 


¢ Have final project statistics been collected and have subjective 
evaluation forms been submitted? Is the software development 
history complete? 


When the management team can answer "Yes" to each question, the 


acceptance testing phase is concluded. 
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SOFTWARE DEVELOPMENT HISTORY 


Material for the development history is collected by the management team throughout the life of 
the project. At the end of the requirements analysis phase, project data and early lessons 
learned are compiled into an initial draft. The draft is expanded and refined at the end of each 
subsequent phase so that, by the end of the project, all relevant material has been collected and 
recorded. The final version of the software development history is produced within 1 month of 


aon 


an | 


software acceptance. The suggested contents are as follows: 


1. Introduction — purpose of system, customer of system, key requirements, development 
machines and language 


2 
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Historical overview by phase — includes products produced, milestones and other key 
events, phase duration, important approaches and decisions, staffing information, and 
special problems 


Requirements definition — if requirements were produced by the software development 
team, this section provides an historical overview of the requirements definition phase. 
Otherwise, it supplies information about the origin and documentation of the system's 
requirements and functional specifications 


. Requirements analysis 


Preliminary design 


|. Detailed design 
. Implementation — coding through integration for each build/release 


System testing 


. Acceptance testing 


Project data 


. Personnel and organizational structure — list of project participants, their roles, and 


organizational affiliation. Includes a description of the duties of each role (e.g., analyst, 
developer, section manager) and a staffing profile over the life of the project 


. Schedule — table of key dates in the development of the project and a chart showing 


each estimate (original plus reestimates at each phase end) vs. actual schedule 

Project characteristics 

(1) Standard tables of the following numbers: subsystems; total, new, and reused 
units; total, new, adapted and reused (verbatim) SLOC, statements, and 
executables; total, managerial, programmer, and support hours; total productivity 

(2) Standard graphs or charts of the following numbers: project growth and change 
histories; development effort by phase; development effort by activity; CPU usage; 
system test profile; error rates; original size estimate plus each reestimate vs. final 
system size; original effort estimate plus each reestimate vs. actual effort required 

(3) Subjective evaluation data for the project — copy of the SEL subjective evaluation 
form (SEF) or report of SEF data from the project data base (see Reference 19) 


Lessons learned — specific lessons and practical, constructive recommendations that 
pinpoint the major strengths and weaknesses of the development process and product, with 
particular attention to factors such as the following: 


. Planning — development plan timeliness and usefulness, adherence to development 


plan, personnel adequacy (number and quality), etc. 


. Requirements — completeness and adequacy for design, change history and stability, 


and clarity (i.e., were there misinterpretations?)}, etc. 
Development — lessons learned in design, code and unit testing 


. Testing — lessons learned in system and acceptance testing 
. Product assurance — adherence to standards and practices; QA and CM lessons learned 


New technology — impact of any new technology used by the project on costs, 
schedules, quality, etc. as viewed by both developers and managers, recommendations 


for future use of the technology 


Figure 9-4. Software Development History Contents 
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SECTION 10 


KEYS TO SUCCESS 
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KEYS TO SOFTWARE DEVELOPMENT SUCCESS 


e Understand your environment 

e Match the process to the environment 

e Experiment to improve the process 

* Don't attempt to use excessively foreign technology 


DOs AND DON'Ts FOR PROJECT 
SUCCESS 


DO... 


* Develop and adhere to a software 
development plan 

e Empower project personnel 

» Minimize the bureaucracy 

* Establish and manage the software 
baseline 

¢ Take periodic snapshots of project 
health and progress 

¢ Reestimate system size, staff effort, 
and schedules regularly 

* Define and manage phase transitions 

* Foster a team spirit 

e Start the project with a small senior 
staff 


DON'T... 


« Allow team members to proceed in 
an undisciplined manner 

* Set unreasonable goals 

- Implement changes without assessing 


their impact and obtaining proper approval 


» Gold plate 

« Overstaff 

« Assume that intermediate schedule 
slippage can be absorbed later 

* Relax standards 

« Assume that a large amount of 
documentation ensures success 
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KEYS TO SOFTWARE DEVELOPMENT SUCCESS 


The recommended approach to software development, described in 
the preceding sections, has been developed and refined over many 
years specifically for the flight dynamics environment. The 
methodology itself is a product of the SEL's learning experience. 
The SEL has found the following to be the keys to success for any 
software development organization. 


Understand the environment. Before defining or tailoring a 
methodology for your project or organization, determine the 
nature of the problem, the limits and capabilities of the staff, and 
the support software and hardware environment. Collect 
baseline measurements of staff effort, system size, development 
schedule, source code growth, software changes and errors, and 
computer usage. 


Match the process to the environment. In planning a project 


or defining a standard methodology, use your understanding of 
the people, environment, and problem domain to select and tailor 
the software process. There must be a match. Be sure that the 
elements of the process have a rationale and can be enforced. If 
you don't intend to or cannot enforce a standard or a procedure, 
do not include it in the plan. Make data collection, analysis, and 
reporting integral parts of the development methodology. 


Experiment to improve the process. Once a comfortable match 
between the process and environment is defined, stretch it, a 
little at a time, to improve- it continuously. Identify candidate 
areas for improvement, experiment with new techniques or 
extensions to the process, and measure the impact. Based on a 
goal of improving the process and/or product, select candidate 
extensions with a high likelihood of success in the environment. 
There should always be an expectation that the change will 
improve the process. Be careful not to change too many things 
at once so that the results from each change can be isolated. Be 
aware that not all changes will lead to improvement, so be 
prepared to back off at key points. 


Don't attempt to use excessively foreign technology. 
Although minor improvements to the standard process can be 
easily shared from one organization to another, carefully 
consider major changes. Do not select and attempt a 
significantly different technology just because it was successful 
in other situations. The technology, and the risk that 
accompanies its adoption, must suit the local environment. 
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Section 10 - Keys to Success 


DOs AND DON'Ts FOR PROJECT SUCCESS 


As standard practice, the SEL records lessons learned on successful 
and unsuccessful projects. The following DOs and DON'Ts, which 
were derived from flight dynamics project experience, are key to 
project success. 


Nine DOs for Project Success 


e Develop and adhere to a software development plan. At the 
beginning of the project, prepare a software development plan 
that sets project goals, specifies the project organization and 
responsibilities, and defines the development methodology that 
will be used, clearly documenting any deviations from standard 
procedures. Include project estimates and their rationale. 
Specify intermediate and end products, product completion and 
acceptance criteria, and mechanisms for accounting progress. 
Identify risks and prepare contingency plans. 


Maintain and use the plan as a "living" document. Record 
updates to project estimates, risks, and approaches at key 
milestones. Provide each team member with the current 
software development plan. Periodically, audit the team for 
adherence to the plan. 


e Empower project personnel. People are a project's most 
important resource. Allow people to contribute fully to the 
project solution. Clearly assign responsibilities and delegate the 
authority to make decisions to specific team members. Provide 
the team with a precise understanding of project goals and 
limitations. Be sure the team understands the development 
methodology and standards to be followed on the project. 
Explain any deviations from the standard development 
methodology to the team. 


e Minimize the bureaucracy. Establish the minimum 
documentation level and meeting schedule necessary to fully 
communicate status and decisions among team members and 
management. Excessive meetings and paperwork slow progress 
without adding value. Don't try to address difficulties by adding 
more meetings and management. More meetings plus more 
documentation plus more management does not equal more 
SUCCESS. 
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Establish and manage the software baseline. Stabilize 
requirements and specifications as early as possible. Keep a 
detailed list of all TBD items — classified by severity of impact 
in terms of size, cost, and schedule — and set priorities for their 
resolution. Assign appropriate personnel to resolve TBD items; 
follow their progress closely to ensure timely resolution. 
Estimate and document the impact to costs and schedules of each 
specifications change. 


Take periodic snapshots of project health and progress, 
replanning when necessary. Compare the project's progress, 
product, and process measures against the project's plan. Also 
compare the current project with similar, past projects and with 
measurement models for the organization. Replan when the 
management team agrees that there is a significant deviation. 
Depending on project goals and limitations, alter the staffing, 
schedule, and/or scope of the project. Do not hesitate to reduce 
the scope of the work when project parameters dictate. 


When the project significantly exceeds detined limits for the 
local environment, stop all project activity, audit the project to 
determine the true project status, and identify problem areas. 
Examine alternatives and devise a recovery plan before 
proceeding again. 


Reestimate system size, staff effort, and schedules regularly. 
As the project progresses, more information is learned about the 
size and complexity of the problem. Requirements change, the 
composition of the development team changes, and problems 
arise. Do not insist on maintaining original estimates. Each 
phase of the life cycle provides new and refined information to 
improve the estimates and to plan the project more effectively. 
There is nothing wrong with realizing that the size has been 
underestimated or that the productivity has been overestimated. 
It is wrong not to be doing something to detect this and take the 
appropriate action. As a rule, system size, effort, and schedule 
should be reestimated monthly. 


Define and manage phase transitions. Much time can be lost 
in the transition from one phase to the next. Several weeks 
before the start of each new phase, review progress to date, and 
set objectives for the next phase. See that the developers receive 
training in the activities of the next phase, and set intermediate 
goals for the team. Clarify any changes that have been made to 
the development plan. While senior developers are finishing up 
the current phase, get junior members of the team started on the 
next phase’s activities. 
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Section 10 - Keys to Success 


Foster a team spirit. Projects may include people from different 
organizations or companies. Maximize commonality and 
minimize differences among project members in all aspects of 
organization and management. Clearly define and communicate 
the roles of individuals and teams, but provide an overall project 
focus. Cross-train as much as possible. Hold combined team 
meetings so that everyone gets the same story. Have everyone 
follow the same rules. Report progress on a project level as well 
as a team level. Struggle through difficulties and celebrate 
successes together us a unit, helping and applauding each other 
along the way. 


Start the project with a small senior staff. A small group of 
experienced senior people, who will be team leaders during the 
remainder of the project, should be involved from the beginning 
to determine the approach to the project, to set priorities and 
organize the work, to establish reasonable schedules, and to 
prepare the software development plan. Ensure that a plan is in 
place before staffing up with junior personnel. 


Eight DON'Ts for Project Success 


Don't allow team members to proceed in an undisciplined 
manner. Developing reliable, high-quality software at low cost 
is not a creative art; it is a disciplined application of a set of 
defined principles, methods, practices, and techniques. Be sure 
the team understands the methodology they are to follow and 
how they are to apply it on the project. Provide training in 
specific methods and techniques. 


Don't set unreasonable goals. Setting unrealistic goals is worse 
than not setting any. Likewise, it is unreasonable to hold project 
personnel to commitments that have become impossible. Either 
of these situations tends to demotivate the team. Work with the 
team to set reasonable, yet challenging, intermediate goals and 
schedules. Success leads to more success. Setting solid reach- 
able goals early in the project usually leads to continued success. 


Don't implement changes without assessing their impact and 
obtaining proper approval. Estimate the cost and schedule 
impact of each change to requirements and specifications, even if 
the project can absorb it. Little changes add up over time. Set 
priorities based on budget and schedule constraints. Explore 
options with the change originator. In cases where changes or 
corrections are proposed during walk-throughs, document the 
proposed changes in the minutes but do not implement them 
until a formal approved specification modification is received. 
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Don't "gold plate". Implement only what is required. Often 
developers and analysts think of additional "little" capabilities or 
changes that would make the system better in their view. Again, 
these little items add up over time and can cause a delay in the 
schedule. In addition, deviations from the approved design may 
not satisfy the requirements. 


Don't overstaff, especially early in the project. Bring people 
onto the project only when there is productive work for them to 
do. A small senior team is best equipped to organize and 
determine the direction of the project at the beginning. 
However, be careful to provide adequate staff for a thorough 
requirements analysis. Early in the project, when there are 
mostly 'soft' products (e.g., requirements analysis and design 
reports), it is often hard to gauge the depth of understanding of 
the team. Be sure the project has enough of the right people to 
get off to a good start. 


Don't assume that an intermediate schedule slippage can be 
absorbed later. Managers and overly optimistic developers tend 
to assume that the team will be more productive later on in a 
phase. The productivity of the team will not change appreciably 
as the project approaches completion of a phase, especially in the 
later development phases when the process is more sequential. 
Since little can be done to compress the schedule during the later 
life cycle phases, adjust the delivery schedule or assign 
additional senior personnel to the project as soon as this problem 
is detected. 


Likewise, don't assume that late pieces of design or code will fit 
into the system with less integration effort than the other parts of 
the system required. The developers’ work will not be of higher 
quality later in the project than it was earlier. 


Don't relax standards in an attempt to reduce costs. Relaxing 
standards in an attempt to meet a near-term deadline tends to 
lower the quality of intermediate products and leads to more 
rework later in the life cycle. It also sends the message to the 
team that schedules are more important than quality. 


Don't assume that a large amount of documentation ensures 
success. Each phase of the life cycle does not necessarily 
require a formally produced document to provide a clear starting 
point for the next phase. Determine the level of formality and 
amount of detail required in the documentation based on the 
project size, life cycle duration, and lifetime of the system. 
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Acronyms 


ACRONYMS 


attitude ground support system 
Assistant Technical Representative 
acceptance test readiness review 
build design review 
computer-aided software enginecring 
configuration control board 
critical design review 
configuration management 
Code Management System 
component origination form 
central processing unit 

change report form 

database management system 
Digital Command Language 
Digital Equipment Corporation 
data flow diagram 

development status form 
functional configuration audit 
Flight Dynamics Facility 
Goddard Space Flight Center 
interface agreement document 


interface control document 


input/output 

Interactive System Productivity Facility 
independent verification and validation 
job control language 

thousand source lines of code 
language-scnsitive editor 

memorandum of information 
memorandum of understanding 
National Aeronautics and Space Administration 
object-oriented design 

personal computer 

physical configuration audit 

project completion statistics form 
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ACRONYMS (cont.) 


program design language (pseudocode) 
preliminary design review 

project estimates form 

personnel resources form 

preliminary system requirements review 
quality assurance 

questions and answers 

review item disposition 

reusable software library 

system concept review 

Software Development Environment 
software developmenymanagement plan 
subjective evaluation form 

Software Engineering Laboratory 
software engineering notebook 
software failure report 

support instrumentation requirements document 
source lines of code 

Software Management Environment 
system and operations concept 

systems Operations requirements document 
software problem report 

system requirements review 

software specifications review 

Systems Technology Laboratory 
software trouble report 

system test readiness review 

to be determined 
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implementation team and 13 
management team and 142 
SEN and 95, 122 

source code and 122 

system test team and 139 


Index 


prototypes and 14 

requirements definition and 6, 114, 137 
requirements definition team managers and 27 
technical representative 13,35, 108 


D 


Dan Bricklin's Demo Program 50 


system testing phase 136, 145 Data 
tools 123 Table 4-1 51 
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CRF see under Forms Table 5-1 77 
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and 32 flow 77 
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configuration audits and 143 acceptance test 10, 91, 116, 129, 144 
CRF and 122, 128 Fig. 7-8 130 
design phases and 16 ‘contents of 142, 155, 157 
errors and 9, 64 exit criteria and 159 
estimates and estimation and 77 requirements and specifications 
"gold-plating” habit of 184 document and 10, 155 
implementation phase 9, 17 system testing phase and 153 
measures and 18 analytical test 91, 111, 116, 119, 129, 
methods and tools of 70, 71, 73, 74, 95, 123, 132, 137, 167 
124 contents of 132 
overly optimistic 184 exit criteria and 159 
PDL and 74,95 build 69, 89, 90, 102, 108, 113, 115 
practice runs and 144 contents of 99, 101 
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prototypes and prototyping 76 build test 
requirements analysis phase 47 Fig. 7-8 130 
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153 implementation 9 
training of 182 integration test 120 
walk-throughs and 75, 87 management see SDMP, hereunder 
Diagrams, design 64, 66, 68, 70, 71, 74, 75, 86, phase 14, 27 
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OOD and 66 reuse 32, 54, 68, 76 
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code inspection 117, 118 exit criteria and 134, 159 
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Fig. 6-3 94 management team and 115 
SEN and 117, 146 requirements and specifications 
test review 120, 121, 147, 153, 176 document and 9 
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Fig. 5-5 81 
updates to 68 
requirements analysis 7, 45, 46, 50, 184 
contents of 54 
Fig. 4-4 54 
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SSR and 59 
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development team and 7, 44, 71 
errors and 149 
exit criteria and 39 
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Fig. 3-2 24 
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reuse and 45 
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system test plan and 9, 146 
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developers and 89, 95 
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contents of 153 
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Documents continued updates to 68, 69, 79, 96, 114, 127, 181, 
draft of 9, 108, 113, 115, 122, 129, 134, 182 
153 Exit criteria see Criteria, exit 
exit criteria and 159 
final version of 10, 141, 168, 176, 177 F 
system testing phase and 137, 153 
Domain analysis see Analysis, domain FCA see under Audits, configuration 
FDF (Flight Dynamics Facility) 1 
E RSL 76 
see also under Environment 
Editor see LSE Forms 
Entry criteria see Criteria, entry COF (component origination form) 120, 126 
Environments CRF (change report form) 122, 145 
acceptance testing 157 developers and 128 
Ada 93 librarian and 122 
constraints of 61, 180 data collection 18 
CPU use and 127 discrepancy report 122, 140, 147, 169, 170 
development team and 101 acceptance test team and 163 
FDF 1, 52, 71, 72, 96, 122, 124, 149, 180 closure of 149 
analysts and 137 development team and 163, 167, 168 
analytical test plans in 132 exit criteria and 159 
SDE 124 management team and 149 
SEL 120, 123 SEL model 152 
Fig. 1-1 1 system test team and 149 
STL 1, 124 PCSF (project completion statistics form) 
test 115, 137, 139, 147 173 
transfer among 124, 142, 180 requirements question-and-answer 47, 48, 50, 
understanding 180 78, 92, 96, 124, 143, 150 
Estimates and estimation 53 RID (review item disposition form) 45, 58 
code 127 notecard 37 
cost 25, 45, 46, 52, 54, 56, 114, 142 ° CDR and 89, 105 
documentation of 181 exit criteria and 39, 61, 82, 105 
effort see staff, hereunder PDR and 68, 80 
measures and 77, 79, 96, 124 requirements definition team and 70,91 
Table 6-1 97 SRR and 39 
performance 44 ` SEF (subjective evaluation form) 173 
performance modeling and 77 test report 121, 147 
requirements 30, 31 
reuse 51, 97, 150 FORTRAN 
schedule 46, 51, 52, 54, 56, 150 FDF environment and 71 
specification modifications and 79, 96, 182, PDL 68, 74 
183 oo preliminary and detailed design phases and 
staff 31, 51, 52, 56, 58, 150 Fig. 5-3 72 
notecard 10 prologs 68, 74 
system 51, 52, 53, 56, 58, 97, 150 see also SAP, under Analyzers 
updates to 79 
TBD requirements and 52, 54, 61 
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Fig. 6-4 98 
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Images, executable see load module, under Builds 
Implementation phase see under Phase 
Inspections 29, 133 
design 92 
Interactive System Productivity Facility see [SPF 
Interfaces 8, 25, 26 
Ada and 72,74 
detailed design phase 92 
errors 74, 117 
exit criteria and 39, 105 
implementation phase 108, 111 
inspection team and 76 
OOD and 71, 72 
PDR and 80 
preliminary design phase 8, 64, 66, 68, 70, 
71, 75 
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smaller projects 35 
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user 101, 119 
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exit criteria and 133 
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system delivery and 176 
tools for 122, 123, 124 
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measures of see Measures 
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releases see Releases 
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Load module see under Builds 

LSE (language-sensitive editor) 95 
PDL and 74 
VAX 123 
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acceptance testing phase 169, 173 
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Table 6-1 97 
recording of 114 
developers responsible for 18 
estimates and estimation as 79 
implementation phase 114, 124, 126 
Table 7-1 125 
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Table 5-1 78 
project histories database 18 
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Table 4-1 51 
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management system see under VAX 
SEL 120 
SEN and 95, 122 
stubs 120 
testing of 110, 111, 120 


N 
(0) 
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PCA see under Audits, configuration 
PCSF see under Forms 
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development team and 68, 89 
documentation of 80, 86, 95, 98, 119 
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methods and tools 70, 91 
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Phase 
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acceptance testing 
acceptance test plan and 155 
entry criteria 159 
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exit criteria 177 
flow diagram Fig. 9-1 163 
products 176 
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design diagrams and 71, 86 
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evaluation criteria 96 
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FORTRAN and 
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Fig. 6-2 88 
measures of 96, 114 
methods and tools 91, 92 
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preliminary design report and 68 
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walk-throughs 92 
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detailed design document and 8 
documentation and 141, 144, 153 
entry criteria 105 
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evaluation criteria 126 
exit criteria 9, 133 
flow diagram Fig. 7-1 109 
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entry criteria 61 
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Fig. 5-5 81 
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transition to 46 
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entry criteria 39 
evaluation criteria 52 
exit criteria 61 
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key activities 44 
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measures of 50, 77 
methods and tools of 47 
products 46, 54 
prototypes and prototyping and 45 
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Phase continued libraries and 16 
reuse analysis and 45 preliminary design phase 64, 69, 71, 80 
walk-throughs 42, 44, 45, 47 Fig. 5-5 81 
requirements definition 22 prototyping plan see under Documents 
evaluation criteria 31 quality assurance and 13, 29, 69, 90, 115 
‘exit criteria 39 requirements analysis phase 54 
flow diagram Fig. 3-1 23 requirements definition phase 32 
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measures of 30 review 27, 42, 46, 92 
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: methods and tools 28 system testing phase 153 
: products 26, 27, 32 tailoring and 11 
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Reviews evaluation 58 
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7 Products implementation phase and 120 
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° development plan and 181, 182 preliminary design phase and 64, 66, 76, 80 
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Quality assurance 13, 69, 74, 80, 90, 93, 102, 
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documents and 141 
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Fig. 7-2 110 
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analysis phase see under Phase 
audits and 143 
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Fig. 6-4 98 
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communication of 90, 114 
CPU time and 127 
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methods and tools for 28 
definition phase see under Phase 
discrepancy and 149 
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defined 48 
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Reviews continued 
build test plan and 129 
format of 
Fig. 7-9 133 
hardcopy materials for 
Fig. 7-10 134 
requirements definition team and 116 
TBD requirements and 126 
CDR (critical design review) 102 
conduct of 8, 86, 89, 90, 113 
exit criteria and 105 
format 
Fig. 6-6 103 
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project size and 12 
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scheduling 46 
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tailoring the life cycle and 12 
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SEL (Software Engineering Laboratory) 50, 71, 
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Documents 
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